# O\_DIRECT and what to do on filesystems that don't support it
**URL:** https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771
**Category:** FoundationDB Core
**Created:** [October 15, 2018, 8:29pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771 "2018-10-15T20:29:05Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@alexmiller](https://forums.foundationdb.org/u/alexmiller)
#### Post date: [October 15, 2018, 8:29pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/1 "2018-10-15T20:29:05Z")
[#274](https://github.com/apple/foundationdb/issues/274) and [#842](https://github.com/apple/foundationdb/issues/842) are variations of the same issue of FDB requiring O\_DIRECT, and some filesystems not supporting the option.
I saw mention that Wavefront has already had to work around this. @killertypo or @mrz, would you happen to be able to clarify what you ended up running internally to work around this?
---
### Author:  [@panghy](https://forums.foundationdb.org/u/panghy)
#### Post date: [October 20, 2018, 8:00am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/2 "2018-10-20T08:00:42Z")
@alexmiller we just did the following:
```
===================================================================
--- flow/AsyncFileKAIO.actor.h
+++ flow/AsyncFileKAIO.actor.h
@@ -35,11 +35,11 @@
open_filename = filename + ".part";
}
- int fd = ::open( open_filename.c_str(), openFlags(flags) | O_DIRECT, mode );
+ int fd = ::open( open_filename.c_str(), openFlags(flags) | O_SYNC, mode );
if (fd<0) {
Error e = errno==ENOENT ? file_not_found() : io_error();
TraceEvent("AsyncFileKAIOOpenFailed").detail("Filename", filename).detailf("Flags", "%x", flags)
- .detailf("OSFlags", "%x", openFlags(flags) | O_DIRECT).detailf("mode", "0%o", mode).error(e).GetLastError();
+ .detailf("OSFlags", "%x", openFlags(flags) | O_SYNC).detailf("mode", "0%o", mode).error(e).GetLastError();
return e;
} else {
TraceEvent("AsyncFileKAIOOpen")
```
That seems to make it happy, this may not the right thing to do of course. 🙂
---
### Author:  [@panghy](https://forums.foundationdb.org/u/panghy)
#### Post date: [October 20, 2018, 8:01am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/3 "2018-10-20T08:01:21Z")
I should stress that we tested it and it works on ZFS but we don’t have any production workloads on ZFS yet.
---
### Author:  [@dave](https://forums.foundationdb.org/u/dave)
#### Post date: [October 20, 2018, 6:43pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/4 "2018-10-20T18:43:56Z")
O\_SYNC really doesn’t make any sense, unless the filesystem is doing something strange with that flag. We call fsync() when we want data to be flushed to disk.
There are two reasons we use O\_DIRECT:
1. Most Linux filesystems don’t properly support kernel async I/O without O\_DIRECT. For example, when a read can be satisfied from the page cache they will block in io\_submit() while copying the data. This prevents keeping multiple I/O requests outstanding and absolutely kills performance. If ZFS supports async I/O properly without O\_DIRECT ([this](https://github.com/zfsonlinux/zfs/pull/2565) seems to imply that it does, and furthermore supports async fsync which would also probably be worth enabling) then this one is a non-issue for you.
2. It’s a waste of memory and memory bandwidth to have two levels of page caching (FDB’s internal cache and the operating system page cache). This could maybe be mitigated to some extent by decreasing the size of one or the other.
There’s also some reason to think that copy-on-write filesystems are not optimal under btrees (it’s another case of several layers doing the same thing, with the btree pager, the filesystem, and the SSD firmware all exposing update-in-place interfaces while doing copy on write underneath, to the detriment of performance and flash lifetime). But this is probably a question for empirical benchmarking.
---
### Author:  [@panghy](https://forums.foundationdb.org/u/panghy)
#### Post date: [October 21, 2018, 12:38am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/5 "2018-10-21T00:38:28Z")
Yeah, O\_SYNC on ext4 would be bad but I am not sure if we saw a huge degradation in performance with O\_SYNC on ZFS (seems like ZFS on O\_SYNC would still flush on every write though so I would assume it would be a lot worse than O\_DIRECT + write\_cache + occasional fsync).
---
### Author:  [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)
#### Post date: [October 22, 2018, 4:14pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/6 "2018-10-22T16:14:10Z")
It looks like an implementation for O\_DIRECT support has been merged recently in ZFS, so we may have something to look forward to there:
[https://github.com/zfsonlinux/zfs/issues/224](https://github.com/zfsonlinux/zfs/issues/224)
---
### Author:  [@atombender](https://forums.foundationdb.org/u/atombender)
#### Post date: [October 22, 2018, 4:42pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/7 "2018-10-22T16:42:33Z")
As mentioned in [my issue](https://github.com/apple/foundationdb/issues/842), requiring `O_DIRECT` is problematic for FUSE volumes (which includes Docker for Mac, possibly Docker for Desktop on Windows). This is in a development environment where performance is _not_ a priority.
---
### Author:  [@alexmiller](https://forums.foundationdb.org/u/alexmiller)
#### Post date: [October 22, 2018, 8:28pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/8 "2018-10-22T20:28:20Z")
> [@atombender](#):
>
> This is in a development environment where performance is _not_ a priority.
If there was an alternative that existed where the only penalty was cache pollution, it opens the door to automatically falling back if `O_DIRECT` is not supported and logging a loud warning in some fashion. That appears to not be the case, so I think you’ll get your flag.
---
### Author:  [@panghy](https://forums.foundationdb.org/u/panghy)
#### Post date: [October 23, 2018, 12:43am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/9 "2018-10-23T00:43:22Z")
Yeah, internally that’s what we have been saying (0.8 for zfs will have O\_DIRECT)
---
### Author:  [@osamarin](https://forums.foundationdb.org/u/osamarin)
#### Post date: [February 13, 2020, 3:06pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/10 "2020-02-13T15:06:28Z")
Are there any plans for supporting block volumes intead of filesystems? They may improve performance and reliability and may be an alternative of file io with O\_DIRECT.
---
### Author:  [@markus.pilman](https://forums.foundationdb.org/u/markus.pilman)
#### Post date: [February 13, 2020, 5:56pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/11 "2020-02-13T17:56:14Z")
Not in the very recent future. Also I don’t think anyone is going to implement this for the current `ssd` storage engine.
However, I believe @SteavedHams was talking about that for Redwood at one time. I don’t know whether there are specific plans.
---
### Author:  [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)
#### Post date: [February 13, 2020, 7:06pm UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/12 "2020-02-13T19:06:35Z")
From what some of us have measured empirically, compared to ext4 and zfs there would be a performance improvement in using a block device directly if for no other reason than io\_submit() tends to block briefly most of the time. This is bad for FDB since it blocks the main thread. Using AIO on a block device directly has the least chance of blocking.
Redwood stores its entire state in a single file, tracks its own free space internally, and reads/writes using a configurable block size, so it should be able to use a block device directly.
While it wouldn’t be too difficult to try this out with some test code, having FDB storage servers use block devices is going to take a lot more effort. Off the top of my head, the FDB worker would no longer be able to:
- get a list of storage engines and their storage server IDs (currently in the filename) on a host
- instance more than one storage engine at a time during storage engine migrations (unless it has multiple block devices available to use but that would be wasteful in the normal case)
---
### Author:  [@osamarin](https://forums.foundationdb.org/u/osamarin)
#### Post date: [February 14, 2020, 7:50am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/13 "2020-02-14T07:50:53Z")
May be some other solutions for storing IDs. Ex - storing a small amount of metadata in the filesystem
while storing the key-value storage in a block device.
There is not a big problem to have several block devices during a maintenance. LVM may be used for this purpuse. It allows to allocate new logical block volumes and delete them. LVM is also useful for making RAIDs, striping, snapshots and so on.
I think the main advantage of block devices is not to waste extra memory for filesystem cache. OS behavior is unpredictable when there is a large iops volume through this cache.
---
### Author:  [@markus.pilman](https://forums.foundationdb.org/u/markus.pilman)
#### Post date: [February 14, 2020, 8:50am UTC](https://forums.foundationdb.org/t/o-direct-and-what-to-do-on-filesystems-that-dont-support-it/771/14 "2020-02-14T08:50:16Z")
> [@osamarin](#):
>
> May be some other solutions for storing IDs. Ex - storing a small amount of metadata in the filesystem
> while storing the key-value storage in a block device.
Yes, that is one way of solving this issue.
> [@osamarin](#):
>
> There is not a big problem to have several block devices during a maintenance. LVM may be used for this purpuse.
LVM itself comes with quite some overhead if you use these features (for snapshots writes will be at least twice as expensive). So you’ll probably rather pay for the overhead of having a filesystem.
> [@osamarin](#):
>
> I think the main advantage of block devices is not to waste extra memory for filesystem cache. OS behavior is unpredictable when there is a large iops volume through this cache.
This is what `O_DIRECT` is giving us, With `O_DIRECT` the file system cache is bypassed. Typically the main benefit of using a block device is that you get more fine-grained control (and can therefore control stuff like fragmentation better). In the days of modern SSDs, I am not convinced the benefits of using a block device warrant the engineering effort required to make this work. Another problem with that is that using block devices make virtualization and containerization harder (by how much I don’t know). So in the work where people want to run FDB in the cloud, this is a serious drawback.