FoundationDB error code 4100 when trying to debug golang program under windows

Using FoundationDB
1

I tried to debug go program which uses go bindings to access FDB in delve (go debugger) and got the following error:
FoundationDB error code 4100 (An internal error occurred)
The 4100 error is returned by fdb_setup_network() C function.
The same program runs without errors and successfully connects to FDB when launched without debugger.
I use fdb client 6.2.25 for windows, go 1.15.2 (buth the same was for 1.13).

What may be the reason for such error and how to avoid it?

2

Unfortunately, a failure in fdb_setup_network is probably early enough that you can’t get trace files, unless perhaps you are calling this function more than once.

I believe it is the case that internal errors should print out some extra details about the error to stderr. In particular, it should have a file and line number of the error as well as a backtrace. Do you get any of this output?

3

@ajbeamon, thank you for the reply, I missed this message.
The whole output of the program is:

dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec testAPI
server listening at: [::]:2345
Internal Error @ c:\dev\cygwin\home\fdb\jenkins\node\workspace\fdbbuildjobs\fdbbuildwindows\work\_build_\src\fdbclient\multiversiontransaction.actor.cpp 1363:

panic: FoundationDB error code 4100 (An internal error occurred)

goroutine 1 [running]:
github.com/apple/foundationdb/bindings/go/src/fdb.MustOpenDefault(0x0)
        C:/Projects/go/src/github.com/apple/foundationdb/bindings/go/src/fdb/fdb.go:258 +0xe8
main.main()
        C:/Projects/go/src/gitlab.spb.openwaygroup.com/platform/go/configservice.git/cmd/test4/main.go:9 +0x36`

The program I executed (I extracted only lines required for test case) is:
package main

import (
	"github.com/apple/foundationdb/bindings/go/src/fdb"
)

func main() {
	fdb.MustAPIVersion(600)
	fdb.MustOpenDefault()
}

The problematic line in fdb source (multiversiontransaction.actor.cpp 1363) is the first ASSERT in the following function:

// ClientInfo
void ClientInfo::loadProtocolVersion() {
	std::string version = api->getClientVersion();
	if(version == "unknown") {
		protocolVersion = ProtocolVersion(0);
		return;
	}

	char *next;
	std::string protocolVersionStr = ClientVersionRef(version).protocolVersion.toString();
	protocolVersion = ProtocolVersion(strtoull(protocolVersionStr.c_str(), &next, 16));

	ASSERT(protocolVersion.version() != 0 && protocolVersion.version() != ULLONG_MAX);
	ASSERT(next == &protocolVersionStr[protocolVersionStr.length()]);
}

I still have no idea why debugger breaks some assert about protocolVersion.

4

Interesting. Are you loading any external client versions? Doing so would require setting the EXTERNAL_CLIENT_DIRECTORY or EXTERNAL_CLIENT_LIBRARY network options.

Assuming not, we could potentially try to read the string being returned by your library from the getClientVersion call. We can call this directly using a function in the C API: fdb_get_client_version, which will return a const char* with the string we want. If you were willing to modify your Go bindings, you could probably add a line something like this:

fmt.Printf("%s\n", C.GoString(C.fdb_get_client_version()))

to the end of the ApiVersion call in fdb.go.

5

@ajbeamon, I do not use multiversion client and there is only one fdb installation on my computer.
The string returned by C.fdb_get_client_version() is the same when I launch the application with or without debugger (delve):
6.2.25,Current version id not currently supported within Windows.,fdb00b062010001

6

I don’t immediately spot anything wrong with that string. I wonder if this is related to https://github.com/apple/foundationdb/issues/3424, which identified an issue that affects ClientVersionRef. This has been fixed on master but not on any prior release, so there’s probably not a great way to confirm this outside of building a patch.

7

I’ve backported the PR addressing the issue I mentioned, which you can see here: https://github.com/apple/foundationdb/pull/3849. It should go into our next release, and hopefully it will resolve your issue.

8

@ajbeamon, thank you!

9

You should now be able to download 6.2.27, which has the patch I described above. If you try it out and still run into problems, let me know.

10

@ajbeamon, thank you!
With 6.2.27 debugger just works.