ARM Support for FoundationDB

(Richard Applebaum) #1

Is ARM support planned for FoundationDB?

(Jesse Bennett) #2

I have at my disposal an ARM Chromebook that I will be willing to try and test. So the main barrier would be the SSE4.2 instructions really right? There are ways to attempt to get around this but not all ARM Archs are equal so I’ll check the processor instruction set. Workarounds exist like here for instance:

This machine runs a Debian kernel and has a project heavily dependent on Boost which I’ve already built against it. Anyhow, here is a screen of the desktop I can’t recommend these machines more for Linux development on ARM they are super quick. Still busy in BSD land but when I get a chance sure why not?

(Richard Applebaum) #3

Interested to see your progress – if you get FDB running on ARM, that opens up a whole lot of possibilities…

(Jesse Bennett) #4

I’ve gotten the green light to go ahead and use this equipment for development. This comes with the stipulation of integrating it with an interactive kiosk. I have some ideas here but should be interesting :grin:

(Geo) #5

Keep up the DataCenter Upheaval work :slight_smile:

(Jesse Bennett) #6

Did a test run last night patching in aarch64 as the architecture. So here is what I’m working with currently on the Chromebook Plus (6 cores detected):

model name : ARMv8 Processor rev 2 (v8l)
BogoMIPS : 48.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 2

OS is Debian Xenial + xfce4:
Linux localhost 4.4.127-13890-g6c206f1711d4 #1 SMP PREEMPT Tue Jun 5 18:02:52 PDT 2018 aarch64 aarch64 aarch64 GNU/Linux

I am chroot-ing into this via crouton. Mono is a dep but found for Mono on ARM it was easier for me to just install Monodevelop because a few C# libraries were missing for some reason this straightened all out. Of course mono-dmcs is required too. Boost and QTCreator are installed already on here when I was working on an original crypto coin which I can only assume now is out there in the wild somewhere.

So after replacing some Intel intrinsics includes with sse2neon here is what happens:

./flow/ThreadPrimitives.h:60:14: error: ‘_mm_pause’ was not declared in this scope

The same with _mm_prefetch and _MM_HINT_T0 as well. So a lot of things are included in sse2neon but some holes are there. I haven’t looked around further very much for substitutes yet. So- it’s pretty cool that the build kicks off and trying to do its thing I think that even getting this far that this is an interesting possibility. Maybe someone who knows embedded better can chime in too. Have a raspberry PI 2 as well which I have access to just lmk. Trying to wrap some stuff up for the day and thought you should at least get an update :thinking:

(Jesse Bennett) #7

Ok, I’ve built it on AARCH64 and tried saving and retrieving some data and it works. Kinda slow to start up but not when operating. I just did it so I’m sure there must be some issues here and there but at least it’s running. I did this one in a pretty sloppy way because I kept expecting to hit a wall and the only one I hit was the client side bindings which don’t understand the .S intrinsics for Intel. I’ll probably fork the build up tomorrow but if you are interested in the changes here is a full diff to look at:

(Jesse Bennett) #8

Alright the repo is up and this code is from a little while back (Jun 8th) so that is pretty recent luckily. Alright, let’s review some of these selections from that last commit.

The CXX flags in the makefile here have some pretty specific flags on them:

CXXFLAGS += -std=c++0x -fpermissive -march=armv8-a+crc+crypto -DARM -D__NEON__ -mcpu=cortex-a72 -DHAVEFP16 -Wno-return-local-addr

-Wno-return-local-addr is probably not needed I just had to add it eventually to the fdbrpc because libcoroutine won’t build without it. Since that was the case didn’t want it coming back to bite me. GCC has AARCH64 flags which are needed only for the ‘crc’ feature really. The ‘crypto’ flag was added because if we decide to use SHA2 or another encryption scheme down the line this is where ARM really shines. For blockchain ARM is used a lot due to its speed just try googling “arm sha256 neon crypto” some time.

I omitted cpuid.h and __cpuid_count in fdbrpc/Platform.cpp for now that one will just have to be tackled later. I think that is a nice one to sandbox and then get something going later on.

Added SSE2NEON under /platform and /fdbrpc for additional support but haven’t delved into what if any impact it has right now I just followed the directions as far as I could for the docs on that project. It says use compiler flag “-mfpu=neon”. My GCC for AARCH64 did not understand this “-mfpu=neon” flag so I tried to force NEON another way there with “-D__NEON__” and whether or not that’s enough I’m not sure yet. In flow/Platform.h xmmintrin.h and x86intrin.h were replaced with an import to that file. Where xmmintrin.h is called checks for the define “__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8” and truthfully I don’t know what that is or if it is defined for this kernel/build for ChromeOS. Remember, I’m using the default Xenial Crouton image for the Chromebook Plus. I found out the processor is called an OP1 and is actually implemented in a few other things. Unfortunately (or fortunately :thinking:), these ARM processors all are very custom its a much different world than Intel.

The function cpu_relax is implemented like so and pretty straighforward:

# define cpu_relax() asm volatile("yield" ::: "memory")

__rtdsc is implemented with it’s counterpart and sort of the same deal except a new int_64_t is used:

int64_t virtual_timer_value;
asm volatile("mrs %0, cntvct_el0" : "=r"(virtual_timer_value));

CRC32 is the fun part actually. When I got here this is where the real test if inlining actually worked took place. Before that the compiler just hummed along as usual. I directly imported arm_neon.h and arm_acle.h here to replace nmmintrin.h in fdbrpc/crc32c.cpp. So, _mm_crc32_u8 is replaced with __crc32cb and _mm_crc32_u32 is replaced with __crc32cw.

In fdbserver/SkipList.cpp _mm_prefetch(function, _MM_HINT_T0) gets replaced with __builtin_prefetch(function). That one was hard to track down however the crypto community really deserves all the credit for finding all these translations for AARCH64 from Intel. You can read this and find it all in one place is where I can contribute here.

Ok, so that is all to building so how about linking? I had to squelch a bunch of stuff here to make it continue and then the build runs into problems getting the bindings to work. Strangely, on this crouton image bash doesn’t know what $CC is? Had to improvise there as well that one was weird and don’t really want to change it system-wide and break the chroot scheme or whatever is going on there. Here is the output:

Assembling     bindings/c/fdb_c.g.S
bindings/c/fdb_c.g.S: Assembler messages:
bindings/c/fdb_c.g.S:1: Error: unknown pseudo-op: `.intel_syntax'
bindings/c/fdb_c.g.S:36: Error: operand 1 should be an integer register -- `mov r11,qword 
bindings/c/fdb_c.g.S:37: Error: operand 1 should be an integer register -- `mov r11,qword ptr[r11]'
bindings/c/fdb_c.g.S:38: Error: unknown mnemonic `jmp' -- `jmp r11'

Looks interesting, if this works then a client would get built. So there is the ARM64 deep dive and everything I encountered there. Maintaining this is mainline would be a pretty big effort with all the ARM variants, I know this now, but if anyone wants to make their own build I think this is a good start, enjoy.

(Christophe Chevalier) #9

@wolfspider I’m currently looking at maybe changing the depency on Mono that is requird to build the flow actor compiler and a few other tools (written in C#) to .NET Core 2.1, and I was wondering if this would help you or make it worse in your attempt at building fdb for ARM platforms.

Discussion: Is there a plan to release the Flow language and SDK?
Prototype PR:

Are you compiling the FoundationDB source on a x86/x64 host, and copying the binaries other to an ARM machine? Or are you building everything from the ARM host itself?

Do you think that having to install the .NET Core SDK on your development machine would be easy? compared to Mono?

If you have some time to spare, could you tell me if running this simple tutorial on your dev host is very easy to do, or if you had to spend more than 10 minutes fiddling with things before running the hello world program?

.NET Core Hello World:

You can select the linux distribution in the dropdown, if you are not running ubuntu


(Christophe Chevalier) #10

By the way, if also looks like you are working on the FreeBSD port, and I would like to know what are your thoughts on requiring the .NET Core SDK on FreeBSD as well.

There seems to be some work going on that front ( and but it still look like a lot of stuff to do, and I don’t think that there are any nightly binaries available yet…

Would building the source inside a linux VM or container be a viable alternative for working on the FreeBSD port, or a non-starter ?

(Jesse Bennett) #11

Ok sure, yeah that sounds like something I’d be interested in doing. I was actually looking at C# support and was having a hard time with Mono by itself so this is something that would definitely help running .Net Core 2.1 instead. So far I have been building this on the ARM machine itself inside a chroot on ChromeOS. So it is running an ARM64 Debian distro. I’ll give this a try and let you know.

(Christophe Chevalier) #12

Looks there are binaries for the .NET SDK for ARM64 in the download page:

So it looks like you could continue building on the ARM64 machine directly.

(Jesse Bennett) #13

Ok, yes that worked once I installed it I got the verification that it installed:

(xenial)ackiosk@localhost:~/Downloads$ dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.301
Commit: 59524873d6

Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-arm64
Base Path: /home/ackiosk/dotnet/sdk/2.1.301/

Host (useful for support):
Version: 2.1.1
Commit: 6985b9f684

.NET Core SDKs installed:
2.1.301 [/home/ackiosk/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.NETCore.App 2.1.1 [/home/ackiosk/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:

Alright thanks, I appreciate it! So to answer the question about FreeBSD as well here yes I do think that they are very close that project I believe got slated for this summer but whether this will be enough I’m not sure. It probably would be best to move to .Net Core either way in my opinion as well. On my MacBook Pro I have .Net Core and Mono both installed so with the Mono stuff for Unity I’ve been trying to convert over to .Net Standard 2.0 for those projects and what I’ve found is that they work just fine but making that transition can be tough on those projects. I was also looking at the C# bindings for FoundationDB and they look really good and can see where they can benefit from the newer features in .Net Core and C# in general. The .Net Core subset has always been stronger when it comes object modeling and networking partly due to EF Core and PowerShell I think being developed along-side .Net Core.