Create your own S2I images

15 | Written on Fri 31 January 2020. Posted in Tutorials | Richard Walker

Another stepping stone towards understanding developing for OpenShift is Source-to-Image. s2i is a tool provided by Red Hat that enables you to include additional information in your source code project, to build a container image and run your code on-demand.

This guide assumes you have done the deckofcards and handofcards coding projects in this series. This guide also builds on from the previous podman exercises.

The Source-to-Image s2i build command has a dependency on a local Docker service. This guide uses RHEL 8 and Podman (no Docker) explicitly. Therefore, there is a subtle difference using the --as-dockerfile option and an extra podman build step.

I'll admit it took some time for me to make sense of Source-to-Image. In a nutshell, it's nothing more than an opinionated approach to building container images.

Starting with a clean base image, s2i builds an application builder image which typically contains standard packages. For example, you might take a minimal RHEL8 base image then create a new image, adding Python, PIP and use PIP to install Django.

This new Python/Django builder image can be then consumed, only needing to add your code and a run script, reducing the time to deploy an application. The less extra stuff to do in the final application image builds, the better.

── Base Image
   └── Builder Image
       └── Application Image

You can think of the process as a basic build pipeline. Most enterprises already have more sophisticated build pipelines in place. A term often used is an "Image Bakery". Final application images ready for consumption in some image registry.

However, Source-to-Image comes out-of-the-box with OpenShift and provides a convenient method for developers to override assemble and run scripts. The scripts, as mentioned earlier, can be included in a source code repository along with source code. Source-to-Image empowers the developer to include additional packages and custom commands while still consuming standard base or custom build images.

S2I process

Environment

This guide uses a fresh installation of Red Hat Enterprise Linux 8 as the base operating system.

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.1 (Ootpa)

Install Podman, Buildah and Skopeo:

sudo dnf install podman buildah skopeo -y
podman --version
podman version 1.4.2-stable3

buildah --version
buildah version 1.9.0 (image-spec 1.0.0, runtime-spec 1.0.0)

skopeo --version
skopeo version 0.1.37

Your client host needs to be registered and attach to a valid subscription for accessing ubi images.

Install Source-to-Image

The s2i tool itself can is a binary that is simply download (https://github.com/openshift/Source-to-Image/releases) and added to your local environment PATH.

sudo yum install wget -y
wget https://github.com/openshift/Source-to-Image/releases/download/v1.2.0/Source-to-Image-v1.2.0-2a579ecd-linux-amd64.tar.gz
tar -xvf Source-to-Image-v1.2.0-2a579ecd-linux-amd64.tar.gz

If not already present, create a bin directory in your users home directory:

mkdir ~/bin

Move the s2i binary in to ~/bin:

mv s2i ~/bin

And refresh you user environment:

source ~/.bashrc
s2i version
s2i v1.2.0

Explore Python UBI

Using valid Red Hat customer portal credentials login to the registry:

podman login registry.redhat.io

Make sure you do have access to registry.redhat.io via proxies or VPN etc.

You can search the registry:

podman search registry.redhat.io/ubi8

And pull base images as needed.

podman pull registry.redhat.io/ubi8/python-36
podman images
REPOSITORY                          TAG      IMAGE ID       CREATED       SIZE
registry.redhat.io/ubi8/python-36   latest   88189ceb8a67   5 weeks ago   825 MB

You could start with the s2i-base image and add, in this case, the needed Python packages. However, as you'll see from the registry search, there are already base images for all the most popular languages.

Our example is concerned with the python-36 image.

You can validate that this is indeed an s2i primed image:

podman inspect --format='{{ index .Config.Labels "io.openshift.s2i.scripts-url"}}' registry.redhat.io/ubi8/python-36

Explore the container:

podman run --name test -it registry.redhat.io/ubi8/python-36 bash

You'll find the following directories and files are present:

/usr/libexec/s2i/assemble
/usr/libexec/s2i/run
/usr/libexec/s2i/usage

And the Python and PIP are installed:

(app-root) python -V
Python 3.6.8
(app-root) pip -V
pip 9.0.3 from /opt/app-root/lib/python3.6/site-packages (python 3.6)

Basic Django example

Starting with a blank canvas, create a new working directory:

mkdir ~/source-to-image 
cd ~/source-to-image 

Use s2i create <image_name> <directory> to start a new project:

s2i create s2i-django s2i-django
cd s2i-django

The command create the following skeleton structure:

.
├── Dockerfile
├── Makefile
├── README.md
├── s2i
│   └── bin
│       ├── assemble
│       ├── run
│       ├── save-artifacts
│       └── usage
└── test
    ├── run
    └── test-app
        └── index.html

You can remove the test directory.

rm -rf test/

The first task is to create your builder image by editing the Dockerfile and including the most common packages required for any subsequent build.

The following Dockerfile is the most basic example. It adds the minimum LABELS used by OpenShift and updates existing packages. Because we're using the python-36 base image, the only addition is the installation of Django, thus making a simple builder image for deploying Django projects.

Example:

vi Dockerfile
FROM ubi8/python-36

LABEL maintainer="Richard Walker <[email protected]>"

LABEL io.k8s.description="Platform for building Django applications" \
      io.k8s.display-name="builder 0.0.1" \
      io.openshift.expose-services="8000:http" \
      io.openshift.tags="builder,python,python36"

USER root

RUN yum -y update && \
    yum clean all
RUN pip install --upgrade pip
RUN pip3 install django

COPY ./s2i/bin/ /usr/libexec/s2i

RUN chown -R 1001:1001 /opt/app-root

USER 1001

EXPOSE 8000

CMD ["/usr/libexec/s2i/usage"]

Any tasks added to the assemble script execute for subsequent application builds, done every time you deploy code using this image. Therefore keeping them to a minimum reduces the build time.

For example, say an application you have created needs some additional packages. Our deckofcards example requires the djangorestframework package. You could add the following to the assemble script:

vi s2i/bin/assemble 
...
echo "---> Building application from source..."

pip install djangorestframework

Or because we also know in our handofcards example, it needs Django, requests and psycopg2-binary we might be happy to include all four dependencies at this stage. Or maybe not? We know djangorestframework has a dependency on Django. So the only common package is in fact Django.

In that case, the deckofcards project would need to include its own assemble script to include djangorestframework. And the handofcards project would need its own assemble script to install both requests and psycopg2-binary.

Confused?

There is no right or wrong place to do this stuff. Source-to-Image provides flexibility. The more specific you prepare a builder image for your applications, the less time to build and deploy your code in OpenShift.

It might also make sense to make a builder image specifically for each application.

OK, for our purposes of demonstration and learning, we'll make a single image with the deckofcards project in mind:

vi s2i/bin/assemble 
...
echo "---> Building application from source..."

pip install djangorestframework

And in this case, we want to keep things simple and keep using the Python development server to run the application, so add the command to the run script:

vi s2i/bin/run 

It's important to include the 0.0.0.0 for it to work within a container:

...
python manage.py runserver 0.0.0.0:8000

With the Dockerfile, assemble and run scripts in place, build a new builder base image. Remember your local system needs to be correctly registered and attached for the yum/dnf commands to work during the build:

buildah bud -t richardwalker.dev/django-s2i-base-img .

List and view the new image:

podman images
REPOSITORY                          TAG      IMAGE ID       CREATED         SIZE
richardwalker.dev/django-s2i-base   latest   dcc66f9625e0   8 minutes ago   905 MB

We now use this new builder base image to build your final application image on-demand.

Time to stop and consider that the next steps using the s2i CLI tool are essentially demonstrating what OpenShift does under the hood when using Source-To-Image as a deployment strategy. This guide is mainly showing you how to develop custom S2I images.

We use an s2i command to create a new Dockerfile, specifying both the builder image to use and location of the source code.

Create and move into a separate directory as not to overwrite the existing Dockerfile:

mkdir doc-build
cd doc-build

The following build command can be done using either code directly from a Git repository or a file system location:

s2i build https://github.com/richardwalkerdev/deckofcards.git richardwalker.dev/django-s2i-base-img deckofcards-s2i-img --as-dockerfile Dockerfile

or

s2i build code/ richardwalker.dev/django-s2i-base-img deckofcards-s2i-img --as-dockerfile Dockerfile

It's this step in the process that s2i in the previous OCP3 and on RHEL7 with Docker, would build the final image in one go. Using the newer Podman tooling introduces the --as-dockerfile Dockerfile requirement and thus the need for an additional podman build step coming up.

The previous command gathers all the artefacts needed to build the final application image. In a nutshell, this includes the source code and any script to overwrite (for example run or assemble:

.
├── Dockerfile
├── downloads
└── upload

Content to be copied in to the final image are under the upload directory.

Contents of this Dockerfile:

FROM richardwalker.dev/django-s2i-base
LABEL "io.openshift.s2i.build.commit.author"="Richard Walker <[email protected]>" \
      "io.openshift.s2i.build.commit.date"="Sat Feb 1 09:28:45 2020 +0000" \
      "io.openshift.s2i.build.commit.id"="50934b79c2b27bd8adee027c7fa4e46335db8753" \
      "io.openshift.s2i.build.commit.ref"="master" \
      "io.openshift.s2i.build.commit.message"="removed database settings" \
      "io.openshift.s2i.build.source-location"="https://github.com/richardwalkerdev/deckofcards.git" \
      "io.k8s.display-name"="deckofcards-s2i" \
      "io.openshift.s2i.build.image"="richardwalker.dev/django-s2i-base"
ENV DISABLE_COLLECTSTATIC="1" \
    DISABLE_MIGRATE="1"
USER root
# Copying in source code
COPY upload/src /tmp/src
# Change file ownership to the assemble user. Builder image must support chown command.
RUN chown -R 1001:0 /tmp/src
USER 1001
# Assemble script sourced from builder image based on user input or image metadata.
# If this file does not exist in the image, the build will fail.
RUN /usr/libexec/s2i/assemble
# Run script sourced from builder image based on user input or image metadata.
# If this file does not exist in the image, the build will fail.
CMD /usr/libexec/s2i/run

Examine the FROM image and the COPY, RUN and CMD.

The application container image can now be built:

podman build -t richardwalker.dev/deckofcards-s2i-img .
podman images 
REPOSITORY                          TAG      IMAGE ID       CREATED              SIZE
richardwalker.dev/deckofcards-s2i-img    latest     3ed400d347c3   6 seconds ago       907 MB
richardwalker.dev/django-s2i-base-img    latest     852b31d3d9dc   13 minutes ago      903 MB

Run a container from the image, take note of the port mapping:

podman run --name deckofcards-s2i-ctr -u 1001 -d -p 7000:8000 richardwalker.dev/deckofcards-s2i-img

You can get into the running container to check things out by obtaining it's CONTAINER ID from podman ps, for example:

podman exec -it c2cb8cf364da bash

or using its name:

podman exec -it deckofcards-s2i-ctr bash

Finally, test the application:

curl 127.0.0.1:8000/deck/

Recap

Let's recap and conceptualise the process once again. A "base image" is used to create a "builder image". I'm calling it a "builder image", but really its just a custom base image.

Base images include the most common of requirements. In our case, we use the Python base image because Django is a Python framework. The "Builder image" adds the next most common requirements. The assemble script can include other package installations, and the run script can include extra commands to start and run the application. Using s2i to build the final "application image" has an awareness of any ./s2i/ files and scripts that may or may not be present in the source code repository. If present, s2i uses them to overwrite existing scripts in the "builder image". If absent, it keeps the original scripts provided in the "builder image".

S2I process

The convenience of Source-to-Image is that it puts some control back into the hands of the developer. run and assemble scripts can be included with their source code to override those provided in the builder image. In our django-s2i-base-img image, we added a pip install in assemble and python manage.py runserver in the run file. dnf install qemu-kvm qemu-img libvirt virt-install libvirt-client virt-manager

Because the handofcards is now configured to use PostgreSQL, we can use the same image but include additional database migration commands to the run script. Let's do that next.

Deploying handofcards

Create another working directory:

mkdir ~/source-to-image/s2i-django/hoc-build
cd  ~/source-to-image/s2i-django/hoc-build

Create a directory to copy the source code into:

mkdir code

And copy your source code into it:

cp -R ~/Django/handofcards-master/handofcards-project/* code/

We know that the handofcards project has different package requirements and it need to do the migration commands when starting.

Create a .s2i/bin directory under code/:

mkdir -p code/.s2i/bin

Add an assemble script:

vi code/.s2i/bin/assemble
#!/bin/bash -e

if [[ "$1" == "-h" ]]; then
  exec /usr/libexec/s2i/usage
fi


if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
  echo "---> Restoring build artifacts..."
  mv /tmp/artifacts/. ./
fi

echo "---> Installing application source..."
cp -Rf /tmp/src/. ./
echo "---> Fixing persmissions..."
chmod -R 775 ./

echo "---> Building application from source..."

pip install django
pip install psycopg2-binary
pip install requests

And add a run script:

vi code/.s2i/bin/run
#!/bin/bash -e

python manage.py makemigrations
python manage.py migrate
python manage.py runserver 0.0.0.0:8000

Now we'll use s2i to generate the Dockerfile using our original django-s2i-base-img

s2i build code/ richardwalker.dev/django-s2i-base-img handofcards-s2i-img --as-dockerfile Dockerfile

You'll now see that the previous command included the assemble and run scripts:

ll upload/scripts/
assemble
run

And finally, create the new application image:

podman build -t richardwalker.dev/handofcards-s2i-img .

Testing

Before testing the handofcards-s2i-img image, we'll need our postgres container running.

To start up a postgres container independent of our previous podman pod examples use the following:

podman run --name pgdb-standalone -e POSTGRES_PASSWORD=password -d -v "/home/user/volumes/postgres/django:/var/lib/postgresql/data:z" -p 5432:5432 postgres

I'm demonstrating a subtle difference while running these containers. We've started them up individually, that is, not a member of the same pod. Therefore the IP Address needs to reflect the IP Address of you're localhost:

podman run --name handofcards-s2i-ctr -u 1001 -d -p 8000:8000 -e DECKOFCARDS_URL=http://192.168.122.21:7000/deck/ -e DATABASE_NAME=cardsdb -e DATABASE_USER=dbuser -e DATABASE_PASSWORD=changeme -e DATABASE_HOST=192.168.122.21 -e DATABASE_PORT=5432 richardwalker.dev/handofcards-s2i-img
````

### Generic S2I image

To build a more universal image it might make more sense to utilize `requirements.txt`.

Why not add the following into your base image:

```text
if [[ -f requirements.txt ]]; then
  echo "---> Installing dependencies ..."
  pip install -r requirements.txt
fi

Logging - a side note

It's standard practice to have application direct their logs to standard out. One trick is to add RUN tasks in Dockerfile that creates soft links. For example:

# forward request and error logs to collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log 
RUN ln -sf /dev/stderr /var/log/nginx/error.log

Conclusion

It might be argued that Source-to-Image doesn't do anything particularly special other than stage a series of regular buildah commands.

That said, it does provide a standard approach in how to prepare and use Source-to-Image deployment strategies with OpenShift.

I guess, creating your library of S2I builder images with good standard package inclusion would result in more efficient S2I builds.

I feel having this grounding, at this level helps prepare for better understanding when working with deployments in OpenShift.

Using S2I in OpenShift does have some auto-detection logic, providing source code includes standard features, it can detect if source code is Java or Python, for example, selecting the appropriate s2i base images accordingly.

However, I believe it's better to have sound knowledge of building and to customise your images and be in control of what is happening under the hood.

Next up, it's time to deploy an OpenShift cluster either in AWS or using local CodeReady containers.

Part 6a: Install OpenShift 4 on AWS

Part 6b: CodeReady Containers

COMMENTS