Building on Windows

KrzysFR · May 12, 2018, 1:32pm

I was not able to find a place that specifically address the build process on Windows, so here are my quick notes.

These are from the perspective of a non-C++ developper, so some of these may seem obvious for seasoned C++ devs

I’m using Visual Studio 2017 (15.7) on Win10.

Need to install Boost

download the 1.52.0 .7z or .zip archive from https://sourceforge.net/projects/boost/files/boost/1.52.0/
unzip it and copy it in C:\Program Files\boost_1_52_0\. Please note that the Boost documentation says to copy them to C:\Program Files\boost\boost_1_52_0\ but the .vcxproj in fdb ommit the “\boost” middle folder !
if you still get errors about boost includes, double check the path is correct! Also you need Boost v1.52, not 1.55 or later !

Need to install Python 2.7

Download the Windows x86-64 MSI installed for Python 2.7.x for Windows
It must be installed to the default location of C:\Python27\

Need to install WiX ToolSet for the MSIInstaller project

if you don’t care about building the MSI installer, just unload the project from the solution explorer.
if VS does not want to open the project, you may need to reinstall WiX
download latest from http://wixtoolset.org/releases/
you may also need to install the “Wix Toolset Visual Studio 2017 Extension” (or 2015) if you get warning about project type not supported.

Need to install Java JDK 8

If you don’t care about building the java binding, just unload the project from the solution explorer.
note: You may need to sell your soul to Oracle to accept the agreement, which I was not willing to do.

For syntax highlighting:

To have syntax highlighting of the flow keywords, you need to copy the usertype.dat file located in flow/actorcompiler/ in your Visual Studio installation dir, following the instructions at https://msdn.microsoft.com/en-us/library/zy61y8b8.aspx
ReSharper C++ does not understand the flow/actor syntax by default (as of version 2018.1) so you may need to disable Resharper when working on the code.

First build (in Debug) can be slow (several minutes on a fast machine). Next builds should be faster.
Build output is in the bin\Debug or bin\Release folder.

Quick sanity check: start the fdbcli.exe in the build output folder. If you get "WARNING: One or more of the processes in the cluster is incompatible with this version of fdbcli." this is still a success, it just means that it is connected to your existing installed cluster which is probably a lower version.

KrzysFR · May 14, 2018, 12:21pm

Both Visual Studio and ReSharper are a bit confused by the custom flow syntax (ACTOR, wait ..).

The build works fine, but the code is filled with red squigglies and the “find all reference” code gets confused a lot, which makes exploring the code a bit more difficult.

Update: the issue comes from Resharper C++ which must be disabled

Example of the intellisense going berserk on the code when Resharper is enabled:

How have you solved this for your day-to-day coding? What are you doing to have highlighting and auto completion for code that uses Actors at the IDE level?

ajbeamon · May 14, 2018, 4:08pm

There’s a file flow/actorcompiler.h that gets included which provides some fake definitions for the flow keywords that make visual studio happy:

github.com

apple/foundationdb/blob/56535db6133e7849392050bd74f1a7e3b2f9ee10/flow/actorcompiler.h#L25


* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/


#ifndef POST_ACTOR_COMPILER


#include "flow.h"


// These are for intellisense to do proper type inferring, etc. They are no included at build time.
#ifndef NO_INTELLISENSE
#define ACTOR
#define DESCR
#define state
#define UNCANCELLABLE
#define choose if(1)
#define when(x) for(x;;)
template <class T> T wait( const Future<T>& );
template <class T> T waitNext( const FutureStream<T>& );
#endif

You can see it’s included here in the file you had open:

github.com

apple/foundationdb/blob/56535db6133e7849392050bd74f1a7e3b2f9ee10/fdbclient/ReadYourWrites.actor.cpp#L21


*
*     http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/


#include "flow/actorcompiler.h"
#include "ReadYourWrites.h"
#include "Atomic.h"
#include "DatabaseContext.h"
#include "StatusClient.h"
#include "MonitorLeader.h"


class RYWImpl {
public:
	template<class Iter> static void dump( Iter it ) {
		it.skip(allKeys.begin);

Perhaps there’s something going on with the inclusion of that file or the definitions that it contains?

We also have a file that can be used to define user types for Visual Studio for purposes of syntax highlighting. The file is located here, and can be applied using the instructions from here.

KrzysFR · May 14, 2018, 4:11pm

It was in fact Resharper that does not understand the code. If I disable R#, the warnings go away. Maybe their C++ parser is not able to recognize the custom macro definitions. I’ll try open a ticket with them about that.

KrzysFR · May 14, 2018, 8:24pm

I missed the fact that you needed to install Pythong 2.7 to build the fdbclient (it was already installed on my other dev box). Updated the top comment.

wolfspider · May 16, 2018, 7:14pm

This is just C++ for VS I’m assuming but curious to see if the new LLVM and/or MinGW can also handle these builds on Windows. If you are able to build this without switching the compiler on Windows that’s pretty cool.

KrzysFR · May 28, 2018, 8:42pm

Looks like Resharper 2018.1.1 has some fixes for C++ that solve the intellisense issues:

Now all that remains are a few similar errors that I’m not sure I understand.

One example is in NativeAPI.actor.cpp:

github.com

apple/foundationdb/blob/master/fdbclient/NativeAPI.actor.cpp#L995


				nextIndex = versionString.find(';', index);
				networkOptions.supportedVersions.push_back_deep(networkOptions.supportedVersions.arena(), ClientVersionRef(versionString.substr(index, nextIndex-index)));
				index = nextIndex + 1;
			}


			ASSERT(networkOptions.supportedVersions.size() > 0);


			break;
		}
		case FDBNetworkOptions::ENABLE_SLOW_TASK_PROFILING:
			validateOptionValue(value, false);
			networkOptions.slowTaskProfilingEnabled = true;
			break;
		default:
			break;
	}
}


void setupNetwork(uint64_t transportId, bool useMetrics) {
	if( g_network )
		throw network_already_setup();

Severity	Code	Description	Project	File	Line	Suppression State	Detail Description
Error (active)	E0308	more than one instance of overloaded function "getValue" matches the argument list:
	fdbclient	c:\DATA\Git\GitHub\foundationdb\fdbclient\NativeAPI.actor.cpp	998		            
            function "getValue(const Future<Version> &version, const Key &key, const Database &cx, const TransactionInfo &info, const Reference<TransactionLogInfo> &trLogInfo)"
            function "getValue(Future<Version> version, Key key, Database cx, TransactionInfo info, Reference<TransactionLogInfo> trLogInfo)"
            argument types are: (Future<Version>, const Key, Database, TransactionInfo, Reference<TransactionLogInfo>)

The issue for R# C++ seems to be that getValue is first defined line 995 with all arguments as const&, but then later its implementation in line 1163 does not have const&. Visual Studio seems ok with this, but R# c++ thinks that they are two different methods, and so getValue(...) appears ambigous. In turn, this trips up the wait(....) over getValue as a side effect.

Removing the const& line 995 does make the error go away. Are they needed?

ajbeamon · May 29, 2018, 2:25pm

The flow actor compiler generates methods with const& for all of its arguments, so to forward declare an actor method you must include const&. I also believe it doesn’t allow you to pass arguments by reference to an ACTOR, so you won’t be able to add const& at 1163 either.

rishabh · October 13, 2019, 7:32pm

I am following this post and fdb’s readme to setup foundationdb on Visual Studio 2019 and I am getting an error while building. This is probably some error in my build setup, but googling is not helping me right now, hence posting the error here.

Steps taken:

Downloaded and unzipped boost to C:\Program Files\boost_1_67_0
Installed mono
I don’t need bindings or MSI of fdb, so didn’t follow anything related to them
Using visual sutio, cloned (github) and setup a new project for fdb
Added CMakeSettings.json in project directory with below contents

Capture.JPG664×542 52.7 KB

When I build the project, I get below output:
1> CMake generation started for configuration: ‘x64-Debug’. 1> Command line: “cmd.exe” /c ““C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2019\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\CMAKE\CMake\bin\cmake.exe” -G “Visual Studio 16 2019” -A x64 -DCMAKE_INSTALL_PREFIX:PATH=“C:\Users\Rishabh\Source\Repos\foundationdb\out\install\x64-Debug” -DBOOST_ROOT:STRING=“C:/Program Files/boost_1_67_0” -DCMAKE_CONFIGURATION_TYPES=“Debug” “C:\Users\Rishabh\Source\Repos\foundationdb” 2>&1” 1> Working directory: C:\Users\Rishabh\Source\Repos\foundationdb\out\build\x64-Debug 1> [CMake] – Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.17134. 1> [CMake] – The C compiler identification is MSVC 19.23.28106.4 1> [CMake] – The CXX compiler identification is MSVC 19.23.28106.4 1> [CMake] – The ASM compiler identification is MSVC 1> [CMake] – Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/cl.exe 1> [CMake] – Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/cl.exe 1> [CMake] – Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/cl.exe – works 1> [CMake] – Detecting C compiler ABI info 1> [CMake] – Detecting C compiler ABI info - done 1> [CMake] – Detecting C compile features 1> [CMake] – Detecting C compile features - done 1> [CMake] – Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/cl.exe 1> [CMake] – Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/cl.exe – works 1> [CMake] – Detecting CXX compiler ABI info 1> [CMake] – Detecting CXX compiler ABI info - done 1> [CMake] – Detecting CXX compile features 1> [CMake] – Detecting CXX compile features - done 1> [CMake] – C:/Users/Rishabh/source/repos/foundationdb C:/Users/Rishabh/source/repos/foundationdb/out/build/x64-Debug 1> [CMake] – Looking for pthread.h 1> [CMake] – Looking for pthread.h - not found 1> [CMake] – Found Threads: TRUE 1> [CMake] – Could NOT find LibreSSL, try to set the path to LibreSSL root folder in the system variable LibreSSL_ROOT (missing: LIBRESSL_CRYPTO_LIBRARY LIBRESSL_SSL_LIBRARY LIBRESSL_TLS_LIBRARY LIBRESSL_INCLUDE_DIR) 1> [CMake] – LibreSSL NOT Found - Will compile without TLS Support 1> [CMake] – You can set LibreSSL_ROOT to the LibreSSL install directory to help cmake find it 1> [CMake] – Found JNI: C:/Program Files/Java/jdk1.8.0_25/lib/jawt.lib (Required is at least version “1.8”) 1> [CMake] – Found Java: C:/Program Files/Java/jdk1.8.0_25/bin/java.exe (found suitable version “1.8.0.25”, minimum required is “1.8”) found components: Development 1> [CMake] – Found Python: C:/Python27/python.exe (found version “2.7.6”) found components: Interpreter 1> [CMake] – Could NOT find Virtualenv (missing: VIRTUALENV_EXE) 1> [CMake] – Current git version 3bb0c12ce9c18ce7b334d6fed349d5e39f2c7d60 1> [CMake] – FDB version is 7.0.0 1> [CMake] – FDB package name is 7.0 1> [CMake] – The CSharp compiler identification is Microsoft unknown 9999 1> [CMake] – The CSharp compiler version is 3.3.1 1> [CMake] – Check for working C# compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/Roslyn/csc.exe 1> [CMake] – Check for working C# compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/Roslyn/csc.exe – works 1> [CMake] – Found Boost: C:/Program Files/boost_1_67_0 (found suitable version “1.67.0”, minimum required is “1.67”) 1> [CMake] – Check versions.h on flow_actors 1> [CMake] – Check versions.h on fdbrpc_actors 1> [CMake] – Check versions.h on fdbclient_actors 1> [CMake] – Check versions.h on fdbserver_actors 1> [CMake] – Check versions.h on fdbcli_actors 1> [CMake] – The ASM_MASM compiler identification is MSVC 1> [CMake] – Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/Hostx64/x64/ml64.exe 1> [CMake] – Check versions.h on fdb_flow_actors 1> [CMake] – Check versions.h on fdb_flow_tester_actors 1> [CMake] – Building fat jar to C:/Users/Rishabh/source/repos/foundationdb/out/build/x64-Debug/packages 1> [CMake] – Check versions.h on fdbbackup_actors 1> [CMake] – Use old fdb at C:/Program Files/foundationdb/bin/fdbserver.exe 1> [CMake] – ADDING SIMULATOR TEST 0 SimpleExternalTest …Skipped lines… 1> [CMake] – ADDING SIMULATOR TEST 109 status/single_process_too_many_config_params 1> [CMake] – Could not find WIX installation - try setting WIX_ROOT or the WIX environment variable (missing: WIX_CANDLE WIX_LIGHT) 1> [CMake] CMake Warning at packaging/msi/CMakeLists.txt:43 (message): 1> [CMake] Could NOT find WIX - will not build installer 1> [CMake] 1> [CMake] 1> [CMake] – ========================================= 1> [CMake] – Components Build Overview 1> [CMake] – ========================================= 1> [CMake] – Build Java Bindings: ON 1> [CMake] – Build with TLS support: OFF 1> [CMake] – Build Go bindings: OFF 1> [CMake] – Build Ruby bindings: OFF 1> [CMake] – Build Python sdist (make package): ON 1> [CMake] – Build Documentation (make html): OFF 1> [CMake] – ========================================= 1> [CMake] – CPACK_COMPONENTS_ALL 1> [CMake] – Configuring done 1> [CMake] – Generating done 1> [CMake] – Build files have been written to: C:/Users/Rishabh/source/repos/foundationdb/out/build/x64-Debug 1> [CMake] 1> Extracted includes paths. 1> Extracted CMake variables. 1> Extracted source files and headers. 1> Extracted code model. 1> Error: Object reference not set to an instance of an object.

Microsoft ® C/C++ Optimizing Compiler Version 19.23.28106.4 for x64

CheckIncludeFile.c

cl /c /Zi /W3 /WX- /diagnostics:column /Od /Ob0 /D WIN32 /D _WINDOWS /D “CMAKE_INTDIR=“Debug”” /D _MBCS /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"cmTC_d2c10.dir\Debug\" /Fd"cmTC_d2c10.dir\Debug\vc142.pdb" /Gd /TC /errorReport:queue “C:\Users\Rishabh\source\repos\foundationdb\out\build\x64-Debug\CMakeFiles\CMakeTmp\CheckIncludeFile.c”

C:\Users\Rishabh\source\repos\foundationdb\out\build\x64-Debug\CMakeFiles\CMakeTmp\CheckIncludeFile.c(1,10): fatal error C1083: Cannot open include file: ‘pthread.h’: No such file or directory [C:\Users\Rishabh\source\repos\foundationdb\out\build\x64-Debug\CMakeFiles\CMakeTmp\cmTC_d2c10.vcxproj]

The CSharp compiler identification could not be found in “C:/Users/Rishabh/source/repos/foundationdb/out/build/x64-Debug/CMakeFiles/3.15.19080502-MSVC_2/CompilerIdCSharp/CompilerIdCSharp.csproj”

Checking whether the ASM_MASM compiler is GNU using “–version” did not match “(GNU assembler)|(GCC)|(Free Software Foundation)”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : --version
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is Clang using “–version” did not match “(clang version)”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : --version
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is AppleClang using “–version” did not match “(Apple LLVM version)”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : --version
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is ARMClang using “–version” did not match “armclang”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : --version
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is HP using “-V” did not match “HP C”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : -V
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is Intel using “–version” did not match “(ICC)”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : --version
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is SunPro using “-V” did not match “Sun C”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : -V
MASM : fatal error A1017:missing source filename
Checking whether the ASM_MASM compiler is XL using “-qversion” did not match “XL C”:
Microsoft ® Macro Assembler (x64) Version 14.23.28106.4
Copyright © Microsoft Corporation. All rights reserved.

MASM : warning A4018:invalid command-line option : -qversion
MASM : fatal error A1017:missing source filename

Am I doing something wrong in my boost setup because of which pthread.h is not recognized? Or is this error because of some different reason and boost setup is actually fine?

alexmiller · October 13, 2019, 8:08pm

CheckIncludeFile.c is CMake’s CHECK_INCLUDE_FILE, which looks like it’s internally used by find_package(Threads) to detect if pthreads is supported. You can see it running, failing, and then detecting a valid (win32) threads implementation instead.

This would make it seem to me like it wouldn’t be your actual issue.

This line has hits on google that seem to all point to a variety of issues across various versions of Visual Studio. I’d probably suggest starting from there?