I read your issue comment and thought that I would share my perspective of running this in Docker. Hopefully someone else can add to this with a perspective of using docker-compose or other orchestration tools.
My thoughts below assume that fdbmonitor will not be auto-started at some point in future releases when the packages are installed onto the host system for setup because of its current behavior that leads to the necessity of a hack for container support.
Is it more sensible to run fdbmonitor, or to run fdbserver directly and allow the container to die if fdbserver crashes or dies?
I think that fdbmonitor should be the binary that is started by the ENTRYPOINT instruction in a Dockerfile.
The main reason is that it reduces variability among supported setups. The binary only uses a few hundred KB of resident memory, though sets up user and group privilege separation, auto-reloads the config file (though this is only fully supported on Linux), and emits useful log messages related to configuration.
How should shared cluster files be handled and propagated?
When obtaining the default cluster file from a package installed on the host, the file is owned by the foundationdb user and group, however, Docker users and groups often do not match the system users and groups, and only share UIDs and GIDs.
For example, using the above Dockerfile on an Ubuntu 16.04 host, by default they happened to map to the _apt user and input group. This means that the file’s default location of /etc/foundationdb is not writable, which is required. This suggests that users should be instructed to copy this file to where the container can access it.
In my opinion, I think that a docker directory should be placed somewhere in the repo, along with a script that does the equivalent of the Debian postinst script, as far as creating the initial fdb.cluster file, possibly combined with the make_public.py script, except without it auto-restarting the fdbmonitor process without checking if it’s running.
Having both the default foundationdb.conf and a script to create fdb.cluster and configure one or more coordinator addresses in it for would make this a bit more intuitive. A custom foundationdb.conf can be overlaid using bind mounts or another mutable type of storage.
I mainly suggest adding the make_public.py script because the documentation warns of potential data loss from malformed cluster files. I just took note of the syntax and edited mine manually.
There are many ways that containers can have persistent read and write access to files, and while I use bind mounts, this is not a solution for everyone.
Users of orchestration systems, like Kubernetes, will want to use something that is both mutable and available over a network, like GlusterFS, as some setups may not have access to local storage.
My point here is that thoughtful documentation about this should be fairly generic and possibly mention both the more common bind mounts method, and something more scalable, like GlusterFS.
Along with that, some users, myself included, use docker-proxy for network routing between containers that share the same private network, without binding certain container services to a port on an egress interface.
This is where suggesting the use of make_public.py with a fixed IP makes sense for setting up the fdb.cluster file, though this is where the opinion of someone who uses more automation tools would be helpful, similar to the use of GlusterFS above, as this probably won’t be helpful to them.
With that said, the make_public.py script currently restarts the server without checking if the process is running. In my opinion, it should not trigger a restart unless the fdbmonitor process is already running. It does already have an argument for a path to a custom fdb.cluster file, which could be mentioned along with copying the file to an accessible location.
In the future, perhaps somehow allowing for hostname resolution in the fdb.cluster file would be helpful.
I do not have an opinion on the best way that this would be implemented, though I suggest it because that is the way I regularly reference my container services over their private networks that are controlled by docker-proxy. This issue posted two days ago refers to this functionality.
That was a lot, so putting it all together along with the notes that were already here:
Here is an example Dockerfile, nearly identical to the one posted by @hiroshi, except with variables that can be configured using shell arguments and that are made available within the container. Hopefully the RUN rm ... instruction can be removed soon in a later version:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y curl python
ARG FDB_VERSION=5.1.7
ENV FDB_VERSION=${FDB_VERSION}
ARG DEB_REVISION=1
ENV DEB_REVISION=${DEB_REVISION}
RUN curl -sO https://www.foundationdb.org/downloads/${FDB_VERSION}/ubuntu/installers/foundationdb-clients_${FDB_VERSION}-${DEB_REVISION}_amd64.deb
RUN curl -sO https://www.foundationdb.org/downloads/${FDB_VERSION}/ubuntu/installers/foundationdb-server_${FDB_VERSION}-${DEB_REVISION}_amd64.deb
RUN dpkg -i foundationdb-clients_${FDB_VERSION}-${DEB_REVISION}_amd64.deb foundationdb-server_${FDB_VERSION}-${DEB_REVISION}_amd64.deb
RUN rm -r /var/lib/foundationdb/data/*
ENTRYPOINT /usr/lib/foundationdb/fdbmonitor
The image can be built with this command:
docker build -t foundationdb:5.1.7 ./foundationdb-docker/
This is the step where fdb.cluster should be configured with one or more coordinator IPs, possibly using a script, and where foundationdb.conf would be customized if needed.
I don’t use docker-compose or Swarm, and hopefully more users will help out here eventually. The container can be created and started using this command:
docker run --init -td \
-v ${DOCKER_VOL}/foundationdb/etc:/etc/foundationdb \
-v ${DOCKER_VOL}/foundationdb/lib:/var/lib/foundationdb \
-v ${DOCKER_VOL}/foundationdb/log:/var/log/foundationdb \
-v ${DOCKER_VOL}/foundationdb/plugins:/usr/lib/foundationdb/plugins \
-p 4500:4500 \
--net foundationdb \
--ip 172.19.0.10 \
--restart=unless-stopped \
--name foundationdb \
foundationdb:5.1.7
From the host, or a container, as long as the foundationdb-clients package has been installed, the fdb.cluster file is accessible, and the IP is reachable, the database must now be reconfigured as noted by @alexmiller:
fdbcli -C ${DOCKER_VOL}/foundationdb/etc/fdb.cluster --exec configure new single memory
According to my opinions listed above, the foundationdb-docker/ directory or equivalent would contain something similar to this layout:
make_clusterfile.pymake_public.py (or a combination of both scripts)foundationdb.confDockerfile
Hopefully this will kick off a larger discussion of what can be improved from here and how to tackle the other important problems.
if another problem arise, and share what works.