Configging regions results in unknown error occurred 4000

Using FoundationDB
1

We are trying to configure the regions based on the following regions.json:

"regions":[{
    "datacenters":[{
        "id":"EA1",
        "priority":2,
    },{
        "id":"EA2",
        "priority":0,
        "satellite":1
    }],
    "satellite_redundancy_mode":"one_satellite_double"
},{
    "datacenters":[{
        "id":"US1",
        "priority":1,
    },{
        "id":"US2",
        "priority":0,
        "satellite":1
    }],
    "satellite_redundancy_mode":"one_satellite_double"
}]

And if we use fileconfigure regions.json to apply the configuration, it will throw ERROR: An unknown error occurred (4000). We have confirmed that with an invalid json, it will throw ERROR: Invalid JSON while with a non-exist file it will throw ERROR: File could not be read (1513).
So it looks like there are some errors about the configuration, but we cannot figure it out. As there are really limited documentations about this.

2

That error means we aren’t properly dealing with some error condition internally. We should try to get it working for you, and then we can work on fixing the error message once we know what’s going wrong.

I notice that there are trailing commas after priority in “EA1” and US1". What happens if you remove those?

3

Thanks for your reply. I’ve tried and it didn’t change anything.

4

I think the reason for the unknown error is that this isn’t a valid json document (it can be made one by enclosing everything in {}). I tried doing that and then got an “Invalid JSON” error for the extra commas. When I removed those, I got an internal error for not having configured datacenter IDs on my cluster. It sounds like we have some work to do to polish this command and more clearly document it, but maybe you can try doing the above and seeing how it goes? I’ll also try configuring my cluster with datacenters and see if I can make more progress.

5

Woah, it really works. I cannot believe I just ignored such simple problem. Thank you a lot!
Further, I still meet a new issue:

ERROR: All storage servers must be in one of the known regions
Type `fileconfigure FORCE <FILENAME>' to configure without this check

So I am trying to understand it. Here is the structure of my cluster:

ip                port    cpu%  mem%  iops    net  class          roles                                
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     2     -       0    transaction                                        
                   4501    0     2     -       0    transaction                                        
                   4502    0     2     -       0    stateless                                          
                   4503    0     2     -       0    stateless                                          
                   4504    0     2     -       0    stateless                                          
                   4505    1     19    43      1    storage        storage                             
                   4506    1     20    43      2    storage        storage                             
                   4507    1     19    43      2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     3     -       0    transaction                                        
                   4501    1     4     1112    0    transaction    log                                 
                   4502    0     2     -       0    stateless                                          
                   4503    0     2     -       0    stateless                                          
                   4504    0     3     -       0    stateless                                          
                   4505    2     41    1112    2    storage        storage                             
                   4506    2     42    1106    2    storage        storage                             
                   4507    2     41    1103    1    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    2     4     985     0    transaction    log                                 
                   4501    0     3     -       0    transaction                                        
                   4502    0     2     -       0    stateless                                          
                   4503    0     2     -       0    stateless                                          
                   4504    0     2     -       0    stateless                                          
                   4505    2     42    1073    2    storage        storage                             
                   4506    2     37    1073    2    storage        storage                             
                   4507    2     39    1063    2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     3     -       0    transaction                                        
                   4501    1     4     640     0    transaction    log                                 
                   4503    0     2     -       0    stateless                                          
                   4504    1     3     -       0    stateless                                          
                   4505    1     34    672     2    storage        storage                             
                   4506    2     42    666     2    storage        storage                             
                   4507    2     42    672     1    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     3     -       0    transaction                                        
                   4501    1     3     653     0    transaction    log                                 
                   4503    0     2     -       0    stateless                                          
                   4504    1     3     -       0    stateless                                          
                   4505    2     42    650     2    storage        storage                             
                   4506    1     41    612     1    storage        storage                             
                   4507    2     40    638     1    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    1     4     955     0    transaction    log                                 
                   4501    0     3     -       0    transaction                                        
                   4503    1     3     -       0    stateless                                          
                   4504    0     2     -       0    stateless                                          
                   4505    2     42    811     1    storage        storage                             
                   4506    2     36    846     2    storage        storage                             
                   4507    2     42    852     2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     2     -       0    transaction                                        
                   4501    1     4     72      0    transaction    log                                 
                   4502    0     2     -       0    stateless                                          
                   4503    0     2     -       0    stateless                                          
                   4504    0     2     -       0    stateless                                          
                   4505    1     20    72      1    storage        storage                             
                   4506    1     20    72      2    storage        storage                             
                   4507    1     20    72      2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    0     2     -       0    transaction                                        
                   4501    0     2     -       0    transaction                                        
                   4502    1     2     -       0    stateless      proxy                               
                   4503    1     2     -       0    stateless      proxy                               
                   4504    4     2     -       1    stateless      cluster_controller,data_distributor 
                   4505    1     19    70      2    storage        storage                             
                   4506    1     21    70      2    storage        storage                             
                   4507    1     20    70      2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    1     4     66      0    transaction    log                                 
                   4501    1     4     66      0    transaction    log                                 
                   4502    0     2     -       0    stateless                                          
                   4503    0     2     -       0    stateless                                          
                   4504    0     2     -       0    stateless                                          
                   4505    1     21    66      2    storage        storage                             
                   4506    1     20    66      1    storage        storage                             
                   4507    1     20    66      2    storage        storage                             
----------------  ------  ----  ----  ------  ---  -------------  -------------------------------------
 xxx.xxx.xxx.xxx   4500    1     4     65      0    transaction    log                                 
                   4501    1     4     65      0    transaction    log                                 
                   4502    1     2     -       0    stateless      proxy                               
                   4503    4     2     -       1    stateless      master,ratekeeper                   
                   4504    0     2     -       0    stateless      resolver                            
                   4505    1     20    65      2    storage        storage                             
                   4506    1     20    65      1    storage        storage                             
                   4507    1     20    65      2    storage        storage

And about the locality_dcid:

locality_dcid = EA1
locality_dcid = EA1
locality_dcid = EA1
locality_dcid = EA2
locality_dcid = EA2
locality_dcid = US1
locality_dcid = US1
locality_dcid = US1
locality_dcid = US2
locality_dcid = US2

So I can confirm that all processes belong to one of the known datacenters(“EA1”, “EA2”, “US1” or “US2”). But why the cluster says that?

6

Not sure, I got the same thing but couldn’t explain it. I’ll have to get back to you on that one. I was able to get it to go through by forcing it, though I won’t explicitly recommend that without knowing why it thinks there’s a problem.

7

Thanks for your efforts :slight_smile:
One additional information you may need is that we are using version 6.1.8.

8

I think the problem here is that there are storage servers running in the satellite datacenters, but the expectation is that there won’t be any (storage should only be in primary datacenters). It may be helpful to follow the sequence of steps here for migrating from a single datacenter cluster to one configure with regions, which hopefully will avoid any of these weird cases.

I’m not sure if it’s necessary to set the machine class in satellites, but if you do I think all you’d need are transaction class processes.

9

I’ve created a PR to solve the two issues that caused an unknown error for you and an internal error for me:

10

Thanks a lot! You are always so helpful :slight_smile: