Stuck Approving System

[jglover7]

I have just setup a SystemLink server and installed the client application on another Windows machine. I have put in the SystemLink server address and the client has gone into the "Not Approved" state, waiting for the server to approve. When I navigate to the web server and login with the admin credentials I can see 1 pending system. I try to approve the pending System but it just spins forever not approving.

I'm using Server and Client 2020R1 (both installed last Friday, 6/19/2020). I have checked all the ports and they are open on the network and reachable from my client (I used nmap -p <ports> from my client). Attached are the master logs from an attempt. I assume that this approval process happens solely between the browser and the server and that the client is not involved. Is this correct?

[JoshuaP]

Have you tried rebooting your SystemLink Server since the issue first occurred?  From the log it seems that the Salt Master is failing to communicate with the RabbitMQ bus.  Perhaps the credentials got out of sync.

[jglover7]

Yes, but the same set of lines appeared in the new log. Perhaps there is something wrong with the RabbitMQ configuration?

[JoshuaP]

Open up a web browser locally on the server (Chrome or Edge) to https://localhost:15672/. It should look something like the following.

Then open C:\ProgramData\National Instruments\Skyline\SkylineConfigurations\skyline_localhost.json with a text editor and use the specified User and Password to see if you can log in.  It should look something like the following and will verify that the RabbitMQ web server is up and running.

[jglover7]

I was able to verify it does connect, I was able to login using the password from the aforementioned .json file, but the approvals are still stuck. Is there a particular in which I should start/restart the services for this to work properly? Does the web server need to come up last? I would assume that this would be taken care of by specifying targets of each of the services.

[JoshuaP]

If you click on the Admin tab in the RabbitMQ web interface do you see the minions listed?  If so, you may want to remove them and then try to approve the targets again.

[jglover7]

No, I do not see any minions listed. What should it look like exactly? Which section under Admin should I find them? Also, in my skyline_localhost.json file it has the port listed as "5673". Is this OK or should it be "15672", like i've used to get to the RabbitMQ through the browser? Checking the port requirements it shows we need both 5673 and 15672 for RabbitMQ, but I don't see the Minion at the moment. 

[jglover7]

When I go to look at the salt logs again for the master it seems to be rapidly rolling the log. I'm guessing this is happening because it is crashing and restarting over and over again... has anyone experienced this before?

[jglover7]

I am still unsure what the issue could be. After checking the logs it seems that the issue first appears when trying to perform an addition of a user using the rabbitMQ HTTP API. The error log tells me there is an SSL error. 

I attempted to recreate the issue with my own Python script that added the user using the exact same URL.

import urllib3
from urllib3 import HTTPSConnectionPool
from pprint import pprint
import json

username = 'niskyline'
password = '<mypassword>'
headers = urllib3.util.make_headers(basic_auth=f'{username}:{password}')
pool = HTTPSConnectionPool(
    '<hostname>',
    port=15672,
    maxsize=1,
    cert_reqs='CERT_NONE',
    headers=headers
)
r = pool.request('GET', '/api/users')
print(r.status)
pprint(json.loads(r.data))

add_user_url = '/api/users/minion_HP_ZBook_15_G4--SN-CND82943QX--MAC-10-62-E5-8A-23-2B'
body = json.dumps({
"password_hash": "",
"tags": "management"
})

put_headers = {'Content-Type': 'application/json'}
put_headers.update(pool.headers)
r = pool.request(
    'PUT',
    add_user_url,
    headers=put_headers,
    body=body
)
print(r.status)

Running this script will get me an HTTP 204, or success. I have to ignore the certificate because of where I am running it on, but there are no SSL errors. Rerunning without the put request shows me the other user.

So follow up questions:

1. Is there a particular body I should put?

2. The main error is an SSL error (as far as the logs indicate), so I would assume that the issue is not with the formation of the body or header, but with connecting. Does the RabbitMQ server need the certificates I have generated for the web server? I have generated certificates for the web server because I do not want SSL issues in my browser. Do I need to update the skyline config with it as well?

[jglover7]

More findings: I noticed that the system shows up in the "Managed Systems" section, but continues to remain in the "Pending Systems" section when I attempt approval. RabbitMQ still does not have the system in its User database and SystemLink still thinks it is disconnected. The minion continues to try the master every 10 seconds. Somehow the master is not formulating the HTTP API request properly for RabbitMQ. I was able to formulate it properly using only information found in "C:/ProgramData/National Instruments/Skyline/".

All I did was point to the rabbitmq certificate and use the username and password located in the skyline_localhost.json configuration file.

Upon restarting the SystemLink services it will try again for maybe 10 times, then doesn't output any more log lines.

Here are my HTTPSConnectionPool settings (these work):

headers = urllib3.util.make_headers(basic_auth=f'{username}:{password}')
pool = HTTPSConnectionPool(
    '<hostname>',
    port=15672,
    maxsize=1,
    ca_certs='rabbitmq-server.cer',
    headers=headers
)