Checks
Commit Message
From: Juraj Linkeš <juraj.linkes@pantheon.tech>
The Dockerfile defines development and CI runner images.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
v3:
Remove pexpect.
v2:
This verson updates the dockerfile to instead install poetry using pipx
due to the version of poetry installed using the package repositories of
the distro being out of date, and to conform to documentation on
installing poetry.
This version also adds extra information to the README about the
preference of using SSH keys, and added a way to inject them into the
devcontainer for vscode.
dts/.devcontainer/devcontainer.json | 33 ++++++++++++++
dts/Dockerfile | 38 ++++++++++++++++
dts/README.md | 70 +++++++++++++++++++++++++++++
3 files changed, 141 insertions(+)
create mode 100644 dts/.devcontainer/devcontainer.json
create mode 100644 dts/Dockerfile
create mode 100644 dts/README.md
Comments
> diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json
> new file mode 100644
> index 0000000000..6313cd3ded
> --- /dev/null
> +++ b/dts/.devcontainer/devcontainer.json
> @@ -0,0 +1,33 @@
> +// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
> +// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
> +{
> + "name": "Existing Dockerfile",
> +
> + // Sets the run context to one level up instead of the .devcontainer folder.
> + "context": "..",
> +
> + // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
> + "dockerFile": "../Dockerfile",
> +
> + // Use 'forwardPorts' to make a list of ports inside the container available locally.
> + // "forwardPorts": [],
> +
> + // Uncomment the next line to run commands after the container is created - for example installing curl.
The next line is uncommented, we should update or remove the comment.
> + "postCreateCommand": "poetry install --no-root",
> +
> + "extensions": [
> + "ms-python.vscode-pylance",
> + ]
> +
> + // Uncomment when using a ptrace-based debugger like C++, Go, and Rust
> + // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
> +
> + // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
> + // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
> +
> + // Uncomment to mount your SSH keys into the devcontainer used by vscode.
> + // "mounts": ["source=${localEnv:HOME}/.ssh,destination=/root/.ssh,type=bind,readonly"]
Should this SSH key correspond to the user below?
> +
> + // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
> + // "remoteUser": "vscode"
> +}
<snip>
> diff --git a/dts/README.md b/dts/README.md
> new file mode 100644
> index 0000000000..dc88ec585e
> --- /dev/null
> +++ b/dts/README.md
> @@ -0,0 +1,70 @@
<snip>
> +#### Start docker container with SSH keys
> +
> +```shell
> +docker build --target dev -t dpdk-dts .
> +docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it dpdk-dts bash
We talked about possibly baking the key into the image, but this seems
safer and pretty easy to use.
I understand this is tailored to the lab and I'm thinking about other
possible use cases, but it seems there would only be convoluted ones
(possibly with some extra security considerations, which is generally
not needed for testing purposes) where this doesn't do what we want it
to. I'd say this is good enough.
> +$ poetry install
> +$ poetry shell
> +```
> +
On Fri, Jan 12, 2024 at 5:23 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> > diff --git a/dts/.devcontainer/devcontainer.json
> b/dts/.devcontainer/devcontainer.json
> > new file mode 100644
> > index 0000000000..6313cd3ded
> > --- /dev/null
> > +++ b/dts/.devcontainer/devcontainer.json
> > @@ -0,0 +1,33 @@
> > +// For format details, see https://aka.ms/devcontainer.json. For
> config options, see the README at:
> > +//
> https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
> > +{
> > + "name": "Existing Dockerfile",
> > +
> > + // Sets the run context to one level up instead of the
> .devcontainer folder.
> > + "context": "..",
> > +
> > + // Update the 'dockerFile' property if you aren't using the
> standard 'Dockerfile' filename.
> > + "dockerFile": "../Dockerfile",
> > +
> > + // Use 'forwardPorts' to make a list of ports inside the
> container available locally.
> > + // "forwardPorts": [],
> > +
> > + // Uncomment the next line to run commands after the container
> is created - for example installing curl.
>
> The next line is uncommented, we should update or remove the comment.
>
Good catch, I'll change this.
>
> > + "postCreateCommand": "poetry install --no-root",
> > +
> > + "extensions": [
> > + "ms-python.vscode-pylance",
> > + ]
> > +
> > + // Uncomment when using a ptrace-based debugger like C++, Go,
> and Rust
> > + // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt",
> "seccomp=unconfined" ],
> > +
> > + // Uncomment to use the Docker CLI from inside the container.
> See https://aka.ms/vscode-remote/samples/docker-from-docker.
> > + // "mounts": [
> "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
> > +
> > + // Uncomment to mount your SSH keys into the devcontainer used
> by vscode.
> > + // "mounts":
> ["source=${localEnv:HOME}/.ssh,destination=/root/.ssh,type=bind,readonly"]
>
> Should this SSH key correspond to the user below?
>
On one hand I agree it would be better to unify the two options, but on the
other we don't make a remote user in the docker image so the option below
can't be used anyway. I would be more in favor of just removing the
remoteUser option and leaving this the way it is as that would line up
better with what you can actually do with the container image we provide.
Leaving it as a stub for something that could be done isn't a bad either
though, or I could also add the remote user to the container, but I don't
really see the need for a non-root user for running DTS currently.
>
> > +
> > + // Uncomment to connect as a non-root user if you've added one.
> See https://aka.ms/vscode-remote/containers/non-root.
> > + // "remoteUser": "vscode"
> > +}
>
> <snip>
>
> > diff --git a/dts/README.md b/dts/README.md
> > new file mode 100644
> > index 0000000000..dc88ec585e
> > --- /dev/null
> > +++ b/dts/README.md
> > @@ -0,0 +1,70 @@
> <snip>
> > +#### Start docker container with SSH keys
> > +
> > +```shell
> > +docker build --target dev -t dpdk-dts .
> > +docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it
> dpdk-dts bash
>
> We talked about possibly baking the key into the image, but this seems
> safer and pretty easy to use.
> I understand this is tailored to the lab and I'm thinking about other
> possible use cases, but it seems there would only be convoluted ones
> (possibly with some extra security considerations, which is generally
> not needed for testing purposes) where this doesn't do what we want it
> to. I'd say this is good enough.
>
>
> +$ poetry install
> > +$ poetry shell
> > +```
> > +
>
On Fri, Jan 12, 2024 at 3:59 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
>
>
> On Fri, Jan 12, 2024 at 5:23 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>
>> > diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json
>> > new file mode 100644
>> > index 0000000000..6313cd3ded
>> > --- /dev/null
>> > +++ b/dts/.devcontainer/devcontainer.json
>> > @@ -0,0 +1,33 @@
>> > +// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
>> > +// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
>> > +{
>> > + "name": "Existing Dockerfile",
>> > +
>> > + // Sets the run context to one level up instead of the .devcontainer folder.
>> > + "context": "..",
>> > +
>> > + // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
>> > + "dockerFile": "../Dockerfile",
>> > +
>> > + // Use 'forwardPorts' to make a list of ports inside the container available locally.
>> > + // "forwardPorts": [],
>> > +
>> > + // Uncomment the next line to run commands after the container is created - for example installing curl.
>>
>> The next line is uncommented, we should update or remove the comment.
>
>
> Good catch, I'll change this.
>
>>
>>
>> > + "postCreateCommand": "poetry install --no-root",
>> > +
>> > + "extensions": [
>> > + "ms-python.vscode-pylance",
>> > + ]
>> > +
>> > + // Uncomment when using a ptrace-based debugger like C++, Go, and Rust
>> > + // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
>> > +
>> > + // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
>> > + // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
>> > +
>> > + // Uncomment to mount your SSH keys into the devcontainer used by vscode.
>> > + // "mounts": ["source=${localEnv:HOME}/.ssh,destination=/root/.ssh,type=bind,readonly"]
>>
>> Should this SSH key correspond to the user below?
>
>
> On one hand I agree it would be better to unify the two options, but on the other we don't make a remote user in the docker image so the option below can't be used anyway. I would be more in favor of just removing the remoteUser option and leaving this the way it is as that would line up better with what you can actually do with the container image we provide.
>
> Leaving it as a stub for something that could be done isn't a bad either though, or I could also add the remote user to the container, but I don't really see the need for a non-root user for running DTS currently.
>
Thanks for the explanation. As there is no need for it and adding
things that could be done (in other tools) seems out of scope to me,
let's remove it.
>>
>>
>> > +
>> > + // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
>> > + // "remoteUser": "vscode"
>> > +}
>>
>> <snip>
>>
>> > diff --git a/dts/README.md b/dts/README.md
>> > new file mode 100644
>> > index 0000000000..dc88ec585e
>> > --- /dev/null
>> > +++ b/dts/README.md
>> > @@ -0,0 +1,70 @@
>> <snip>
>> > +#### Start docker container with SSH keys
>> > +
>> > +```shell
>> > +docker build --target dev -t dpdk-dts .
>> > +docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it dpdk-dts bash
>>
>> We talked about possibly baking the key into the image, but this seems
>> safer and pretty easy to use.
>> I understand this is tailored to the lab and I'm thinking about other
>> possible use cases, but it seems there would only be convoluted ones
>> (possibly with some extra security considerations, which is generally
>> not needed for testing purposes) where this doesn't do what we want it
>> to. I'd say this is good enough.
>>
>>
>> > +$ poetry install
>> > +$ poetry shell
>> > +```
>> > +
new file mode 100644
@@ -0,0 +1,33 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
+{
+ "name": "Existing Dockerfile",
+
+ // Sets the run context to one level up instead of the .devcontainer folder.
+ "context": "..",
+
+ // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
+ "dockerFile": "../Dockerfile",
+
+ // Use 'forwardPorts' to make a list of ports inside the container available locally.
+ // "forwardPorts": [],
+
+ // Uncomment the next line to run commands after the container is created - for example installing curl.
+ "postCreateCommand": "poetry install --no-root",
+
+ "extensions": [
+ "ms-python.vscode-pylance",
+ ]
+
+ // Uncomment when using a ptrace-based debugger like C++, Go, and Rust
+ // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
+
+ // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
+ // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
+
+ // Uncomment to mount your SSH keys into the devcontainer used by vscode.
+ // "mounts": ["source=${localEnv:HOME}/.ssh,destination=/root/.ssh,type=bind,readonly"]
+
+ // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
+ // "remoteUser": "vscode"
+}
new file mode 100644
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+
+# There are two Docker images defined in this Dockerfile.
+# One is to be used in CI for automated testing.
+# The other provides a DTS development environment, simplifying Python dependency management.
+
+FROM ubuntu:22.04 AS base
+
+RUN apt-get -y update && apt-get -y upgrade && \
+ apt-get -y install --no-install-recommends \
+ python3 \
+ python3-pip \
+ pipx \
+ python3-cachecontrol \
+ openssh-client && \
+ pipx install poetry>=1.5.1 && pipx ensurepath
+WORKDIR /dpdk/dts
+
+
+FROM base AS runner
+
+# This image is intended to be used as the base for automated systems.
+# It bakes DTS into the image during the build.
+
+COPY . /dpdk/dts
+RUN poetry install --no-dev
+
+CMD ["poetry", "run", "python", "main.py"]
+
+FROM base AS dev
+
+# This image is intended to be used as DTS development environment. It doesn't need C compilation
+# capabilities, only Python dependencies. Once a container mounting DTS using this image is running,
+# the dependencies should be installed using Poetry.
+
+RUN apt-get -y install --no-install-recommends \
+ vim emacs git
new file mode 100644
@@ -0,0 +1,70 @@
+# DTS Environment
+The execution and development environments for DTS are the same, a
+[Docker](https://docs.docker.com/) container defined by our [Dockerfile](./Dockerfile).
+Using a container for the development environment helps with a few things.
+
+1. It helps enforce the boundary between the DTS environment and the TG/SUT, something
+ which caused issues in the past.
+2. It makes creating containers to run DTS inside automated tooling much easier, since
+ they can be based off of a known-working environment that will be updated as DTS is.
+3. It abstracts DTS from the server it is running on. This means that the bare-metal os
+ can be whatever corporate policy or your personal preferences dictate, and DTS does
+ not have to try to support all distros that are supported by DPDK CI.
+4. It makes automated testing for DTS easier, since new dependencies can be sent in with
+ the patches.
+5. It fixes the issue of undocumented dependencies, where some test suites require
+ python libraries that are not installed.
+6. Allows everyone to use the same python version easily, even if they are using a
+ distribution or Windows with out-of-date packages.
+7. Allows you to run the tester on Windows while developing via Docker for Windows.
+
+## Tips for setting up a development environment
+
+### Getting a docker shell
+These commands will give you a bash shell inside the container with all the python
+dependencies installed. This will place you inside a python virtual environment. DTS is
+mounted via a volume, which is essentially a symlink from the host to the container.
+This enables you to edit and run inside the container and then delete the container when
+you are done, keeping your work. It is also strongly recommended that you mount your SSH
+keys into the container to allow you to connect to hosts without specifying a password.
+
+#### Start docker container with SSH keys
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it dpdk-dts bash
+$ poetry install
+$ poetry shell
+```
+
+#### Start docker container without SSH keys
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd)/..:/dpdk -it dpdk-dts bash
+$ poetry install
+$ poetry shell
+```
+
+### Vim/Emacs
+Any editor in the ubuntu repos should be easy to use, with vim and emacs already
+installed. You can add your normal config files as a volume, enabling you to use your
+preferred settings.
+
+```shell
+docker run -v ${HOME}/.vimrc:/root/.vimrc -v $(pwd)/..:/dpdk -it dpdk-dts bash
+```
+
+### Visual Studio Code
+VSCode has first-class support for developing with containers. You may need to run the
+non-docker setup commands in the integrated terminal. DTS contains a .devcontainer
+config, so if you open the folder in vscode it should prompt you to use the dev
+container assuming you have the plugin installed. Please refer to
+[VS Development Containers Docs](https://code.visualstudio.com/docs/remote/containers)
+to set it all up. Additionally, there is a line in `.devcontainer/devcontainer.json`
+that, when included, will mount the SSH keys of the user currently running vscode into
+the container for you. The `source` on this line can be altered to mount any SSH keys on
+the local machine into the container at the correct location.
+
+### Other
+Searching for '$IDE dev containers' will probably lead you in the right direction.