Need Foundationdb tools for performance monitor


(VISHAL AGGARWAL) #1

Apart from fdbcli json status details , Is there any foundationdb/document layer tool I can download to monitor read/write transaction ops per second . I was going through some of the forum mention about fdbtop and fdbshell but have no idea how to download them to use it in Red hat linux .

Any help will be appreciated …


(Christophe Chevalier) #2

FdbTop and FdbShell are build using .NET but can run on linux using .NET Core. I haven’t build a full package yet, but you can build it manually. I’ll try to take some time to create a windows and linux release for these tools.

Manual build instructions

  • Install .NET Core SDK 2.2 on your linux workstation (https://dotnet.microsoft.com/download/linux-package-manager/rhel/sdk-current) and ensure that dotnet --version does give you version 2.2.something.
  • Clone the .NET binding repository (https://github.com/Doxense/foundationdb-dotnet-client):
    • git clone https://github.com/Doxense/foundationdb-dotnet-client.git
  • from the root of the repository, build and publish it for your platform:
    • dotnet publish -c Release -f netcoreapp2.2 -r linux-x64 FdbTop -o ../build
  • go to the build subfolder, and you should see all the .NET Core runtime, dll and an FdbTop native linux application.
    • run ./FdbTop and it should work

The build folder contains a native linux build that you can then zip and deploy on any other RHEL/Ubuntu/XYZ distribution without having to install the .NET Core runtime.

For people wanting to build the same thing for windows, replace linux-x64 by win10-x64 to produce a FdbTop.exe.

Some notes:

  • you must have installed the FoundationDB Client package on your machine, and fdbcli should already work.
  • your terminal size must be at least 136x30 because I haven’t written a “responsive” UI in text mode :slight_smile:
  • you can specify a cluster file by running ./FdbTop path/to/foo.cluster

To build FdbShell, it is the same process, replacing FdbTop by FdbShell in the dotnet publish step.

Following these instructions on an Ubuntu 18.10 I got it running:


(Christophe Chevalier) #3

Oh and also, there is another tool called fdbtop that you can find here: https://github.com/poma/fdbtop

It’s written in javascript, but it seems to be calling fdbcli directly so I don’t think that it requires the node.js binding installed.


(VISHAL AGGARWAL) #4

Thanks for the information ,
I want to try on my mac first fdbtop and fdbshell before I install on linux machine
Regarding fdbshell for mac

  • dotnet publish -c Release -f netcoreapp2.2 -r linux-x64 FdbTop -o ../build

what should I type in for it to build for mac
I tried dotnet publish -c Release -f netcoreapp2.2 -r macos FdbTop -o …/build but errored

Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj...
  Restoring packages for /src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj...
  Installing System.Runtime.CompilerServices.Unsafe 4.5.2.
  Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/obj/FoundationDB.Client.csproj.nuget.g.props.
  Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/obj/FoundationDB.Client.csproj.nuget.g.targets.
  Restore completed in 933.54 ms for /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj.
  Installing Microsoft.NETCore.DotNetAppHost 2.2.2.
  Installing Microsoft.NETCore.DotNetHostResolver 2.2.2.
  Installing Microsoft.NETCore.DotNetHostPolicy 2.2.2.
  Installing Microsoft.NETCore.App 2.2.2.
  Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FdbTop/obj/FdbTop.csproj.nuget.g.props.
  Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FdbTop/obj/FdbTop.csproj.nuget.g.targets.
  Restore completed in 2.6 sec for /src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj.
/src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj : error : NETSDK1056: Project is targeting runtime 'macos' but did not resolve any runtime-specific packages. This runtime may not be supported by the target framework.

-Vishal


(VISHAL AGGARWAL) #5

I used osx.10.11-x64 instead and it worked for fdbtop
But for fdbshell it is giving me error

Vishals-MacBook-Pro:foundationdb-dotnet-client root# dotnet publish -c Release -f netcoreapp2.2 -r osx.10.11-x64 FdbTop -o ../build 

Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core

Copyright (C) Microsoft Corporation. All rights reserved.

Restoring packages for /src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj...

Restoring packages for /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj...

Restore completed in 167.3 ms for /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj.

Installing runtime.osx-x64.Microsoft.NETCore.DotNetAppHost 2.2.2.

Installing runtime.osx-x64.Microsoft.NETCore.DotNetHostResolver 2.2.2.

Installing runtime.osx-x64.Microsoft.NETCore.DotNetHostPolicy 2.2.2.

Installing runtime.osx-x64.Microsoft.NETCore.App 2.2.2.

Restore completed in 6.03 sec for /src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj.

Layers/Directories/FdbDirectoryExtensions.cs(283,1): warning CS1570: XML comment has badly formed XML -- 'Expected an end tag for element 'summary'.' [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Status/FdbSystemStatus.cs(768,23): warning CS0168: The variable 'pm' is declared but never used [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Fdb.Options.cs(172,5): warning CS0618: 'Fdb.Options.TLSPlugin' is obsolete: 'This option is deprecated since v6.0' [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Fdb.Options.cs(187,5): warning CS0618: 'Fdb.Options.TLSPlugin' is obsolete: 'This option is deprecated since v6.0' [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Subspaces/DynamicKeySubspace.cs(188,86): warning CS0419: Ambiguous reference in cref attribute: 'Pack{TTuple}'. Assuming 'DynamicKeys.Pack<TTuple>(TTuple)', but could have also matched other overloads including 'DynamicKeys.Pack<T1>(ValueTuple<T1>)'. [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Layers/Directories/FdbDirectoryExtensions.cs(282,3): warning CS1587: XML comment is not placed on a valid language element [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Tuples/STuple.cs(661,66): warning CS1573: Parameter 'tuple' has no matching param tag in the XML comment for 'STuple.Deformatter.ParseNext(string, out IVarTuple, out string)' (but other parameters do) [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Tuples/STuple.cs(661,84): warning CS1573: Parameter 'tail' has no matching param tag in the XML comment for 'STuple.Deformatter.ParseNext(string, out IVarTuple, out string)' (but other parameters do) [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Memory/Slice.cs(51,65): warning CS1574: XML comment has cref attribute 'ReadOnlySpan{T}' that could not be resolved [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Memory/Slice.Encoding.cs(1458,77): warning CS0419: Ambiguous reference in cref attribute: 'FromString'. Assuming 'Slice.FromString(string)', but could have also matched other overloads including 'Slice.FromString(char*, int, ref byte[])'. [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Memory/Slice.Encoding.cs(1458,141): warning CS0419: Ambiguous reference in cref attribute: 'FromStringUtf8WithBom'. Assuming 'Slice.FromStringUtf8WithBom(string)', but could have also matched other overloads including 'Slice.FromStringUtf8WithBom(char*, int, ref byte[])'. [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

Shared/Memory/Slice.Encoding.cs(1458,178): warning CS0419: Ambiguous reference in cref attribute: 'FromByteString'. Assuming 'Slice.FromByteString(string)', but could have also matched other overloads including 'Slice.FromByteString(char*, int, ref byte[])'. [/src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj]

FoundationDB.Client -> /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/bin/Release/netstandard2.0/FoundationDB.Client.dll

Program.cs(182,25): warning CS0168: The variable 'e' is declared but never used [/src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj]

Program.cs(315,26): warning CS0168: The variable 'e' is declared but never used [/src/github.com/foundationdb-dotnet-client/FdbTop/FdbTop.csproj]

FdbTop -> /src/github.com/foundationdb-dotnet-client/FdbTop/bin/Release/netcoreapp2.2/osx.10.11-x64/FdbTop.dll

FdbTop -> /src/github.com/foundationdb-dotnet-client/build/

Vishals-MacBook-Pro:foundationdb-dotnet-client root# 

Vishals-MacBook-Pro:foundationdb-dotnet-client root# dotnet publish -c Release -f netcoreapp2.2 -r osx.10.11-x64 FdbShell -o ../build

Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core

Copyright (C) Microsoft Corporation. All rights reserved.

Restoring packages for /src/github.com/foundationdb-dotnet-client/FdbShell/FdbShell.csproj...

Restore completed in 20.64 ms for /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/FoundationDB.Client.csproj.

Installing ILRepack.MSBuild.Task 1.0.9.

Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FdbShell/obj/FdbShell.csproj.nuget.g.props.

Generating MSBuild file /src/github.com/foundationdb-dotnet-client/FdbShell/obj/FdbShell.csproj.nuget.g.targets.

Restore completed in 1.22 sec for /src/github.com/foundationdb-dotnet-client/FdbShell/FdbShell.csproj.

FoundationDB.Client -> /src/github.com/foundationdb-dotnet-client/FoundationDB.Client/bin/Release/netstandard2.0/FoundationDB.Client.dll

Program.cs(169,21): warning CS0168: The variable 'e' is declared but never used [/src/github.com/foundationdb-dotnet-client/FdbShell/FdbShell.csproj]

FdbShell -> /src/github.com/foundationdb-dotnet-client/FdbShell/bin/Release/netcoreapp2.2/osx.10.11-x64/FdbShell.dll

FdbShell -> /src/github.com/foundationdb-dotnet-client/build/

(VISHAL AGGARWAL) #6

Guys, I was able to download fdbshell , Any specific commands to see the key value pair details or others

Vishals-MacBook-Pro:build root# ./FdbShell

Connecting to cluster…

Using API v600 (max 600)

Cluster file: <default>

FoundationDB Shell menu:

cd Change the current directory

dir List the sub-directories the current directory

show Show the content of the current directory

tree Show all the directories under the current directory

sampling Display statistics on random shards from the database

coordinators Show the current coordinators for the cluster

quit Quit

Ready…

[fdb:cluster_fdb /]# show


(Christophe Chevalier) #7

FdbShell is meant for database that use the Directory Layer and keys that use the Tuple Layer encoding.

Use dir to list directory subspaces, cd to change current path, and show to watch content of the current subspace (assuming it has keys). You can use tree to recursively list all subfolders.

If you don’t see anything, either the database is empty, or does not use the Directory Layer.


(VISHAL AGGARWAL) #8

Thanks Christophe Chevalier


(Bhaskar Muppana) #9

Document Layer doesn’t use Tuple layer encoding for documents. But, it uses the directory layer. You can use the commands @KrzysFR suggested to list directories and all. But, you may not want to use it read the documents. For that, I suggest using standard MongoDB tools like mongo shell or PyMongo.


(Christophe Chevalier) #10

If the keys don’t look like tuple, and the values don’t look like text or JSON, then they will be displayed as hexa-encoded blobs, limiting the readability (unless you are fluent in protobuf or msgpack :slight_smile: ).

Maybe in v2 of the Directory Layer, there will be a way to encode the dialect of a directory subspace.


(matthew zeier) #11

The follow on posts here have some point.

And my post here using Wavefront is an excellent way to performance monitor Fdb.


(VISHAL AGGARWAL) #12

Can you provide some inside on how to install wavefront and configure with FDB

Thanks