How Siteminder Agent interacts with Policy Server

 I found this below discussion in one of the forums and Hope this will be useful in understanding the flow,
 Discussion 1:
============
1.When user enters the application url in the browser, it goes to the web server, if web agent is installed on the web server, The request is first intercepted by the webagent.conf file
2. In web agent.conf file it finds the path of the smhost.conf file and the request goes to smhost.conf file.
3.In smhost.conf file it finds the policy server which it needs to contact.
4.Now Web server contacts the policy server through the ports 4444(1,2,3) that are mentioned in smhost.conf file. This is done by LLAWP(Low Level Agent Worker Process)
5.LLAWP sends the request to policy server asking for initializing the communication between the web agent and policy server. In the request web agents send the details of trusted host that is present in smhost.conf file
6.When policy server receives the request it checks whether it can trust the request by seeing whether the trusted host present is present in policy store and if it belongs to web server from which it got the request.
7. This initial request sent by web agent is encrypted by the shared secret present in smhost file. So the request is securely and only policy server can decry-pt it.
8. If policy server trusts the request it starts the communication with the web agent. This communication will be started HLA process from policy server. HLA process uses HCO that web agent used to register with policy server.
9. When the communication from policy server reaches the web agent it decrypts the communication using shared secret and in communication it will have details about the HCO which have started the communication, it compares if HCO in communication is same as HCO present in smhost.conf file.
10.This way the policy server and web agent handshake with each other and sets the communication.Now Web agent sends the details of the ACO present in webagent.conf to policy server.
11.The policy server now takes the this ACO and it will see if it is present in policy store, if it is present it will get the agent name present in ACO and also the properties of the agent present in ACO.
12. Now policy server asks for the resource and web agents sends the resource it go from browser.
13. policy server gets the list of all the realms present in policy store which are linked to the agent that is present in the ACO. It compares the resource it got from web agent is present in any realm in the list (Here it sees the longest match) and gets the realm which has this resource and policy server see's whether this realm is configured for protected or not protected. If not protected , it will say web agent it is not protected.
14.if it is protected, it will whether the request that got from browser has smsession. If no smsession then policy server fetches the authscheme from realm and gives the corresponding login page to web agent and it gives it to browser.
This the process that will occur to check whether the resource or request got from browser is protected or not.
Note: All the communication that occur after hand shake between policy server and web agent is in encrypted for security. The encryption and decryption is done using keys present in key store.

Discussion 2:
=========
WebAgent Initialization:
========================

When you register the agent with the policy server thru smreghost command, The SmHost.conf file gets created and SmHost.conf file contains information that the Web Agent uses to make initial connections to the Policy Servers to which it is associated.it also creates a trustedHost object into the policy store.which you specify at the time of registration.
Trusted Host is a client computer where one or more Web Agents is installed. It handles the connection to the Policy Server. The term trusted host refers to the physical system. You can have more than one trusted host on a physical server, but each must be identified by a unique name.

Now when you enable the webagent and bounce the webserver, the initial connection flow goes like this, the webagent reads the WebAgent.conf file and looks for the path of SmHost.conf file, from that path it reads the SmHost.conf file and tries to connect to the IP Address of the policy Server m/c which are present there. once that is done, it then fetches the trustedHost object from the policy store (The policy Store is connected with the policy Server). Once that is done then a successful TCP connection is established between the webAgent and the policy server. the WebAgent then never uses the SmHost.conf file.


Before i expalin the request processing part, let me explain the connection between WebAgent and the WebServer. There is a Http connection between the WebAgent and the WebServer. As soon as you configure the webagent with the webserver, remember we specify the webagent binaries inside the configuration of webserver. so when you bounce the webserver after configuration, it loads the webagent DLL / Binaries into the webserver process. This is how a Http connection is made.

Request Processing:
===================
Now when user access a resource from the browser say http://www.sample.com/mysite/abc.html . The WebAgent filters this request from the web server module and after parsing the host and the resource name webagent sends the resource name to the policy server thru the TCP socket connection in form of packets. for e.g. ACO, AgentName, HostName, Client IP, resourceName etc. each in a seperate encrypted packet.
Now Policy Server decrypts the each packet and it knows the agent name which has sent the request, from the agent name it fetches all the matching Realm Object and from the Realm object it checks the resource which has been associated with a specific realm, it then checks that if the resource is protected in a realm and if it is protected, it sends a challange response back to the webagent over the same TCP socket connection, WegAgent who is listing over that socket connection reads the response and throw a chalange to the user in the browser.
This is how a basic IsProtected request is processed.


One thing i want to mention that each request goes seperately, IsProtected request is different then IsAuthenticated Request is different and IsAuthorize request is different.
There are three different requests.

SSL Handshake

The Secure Sockets Layer (SSL) protocol uses a combination of public-key and symmetric-key encryption. Symmetric-key encryption is much faster than public-key encryption; however, public-key encryption provides better authentication techniques. An SSL session always begins with an exchange of messages called the SSL handshake. The handshake allows the server to authenticate itself to the client by using public-key techniques, and then allows the client and the server to cooperate in the creation of symmetric keys used for rapid encryption, decryption, and tamper detection during the session that follows. Optionally, the handshake also allows the client to authenticate itself to the server.

The steps involved in the SSL handshake are as follows (note that the following steps assume the use of the cipher suites listed in Cipher Suites with RSA Key Exchange: Triple DES, RC4, RC2, DES): 

1. The client sends the server the client's SSL version number, cipher settings, session-specific data, and other information that the server needs to communicate with the client using SSL.
2. The server sends the client the server's SSL version number, cipher settings, session-specific data, and other information that the client needs to communicate with the server over SSL. The server also sends its own certificate, and if the client is requesting a server resource that requires client authentication, the server requests the client's certificate.
3. The client uses the information sent by the server to authenticate the server (see Server Authentication for details). If the server cannot be authenticated, the user is warned of the problem and informed that an encrypted and authenticated connection cannot be established. If the server can be successfully authenticated, the client proceeds to step 4.
4. Using all data generated in the handshake thus far, the client (with the cooperation of the server, depending on the cipher being used) creates the pre-master secret for the session, encrypts it with the server's public key (obtained from the server's certificate, sent in step 2), and then sends the encrypted pre-master secret to the server.
5. If the server has requested client authentication (an optional step in the handshake), the client also signs another piece of data that is unique to this handshake and known by both the client and server. In this case, the client sends both the signed data and the client's own certificate to the server along with the encrypted pre-master secret.
6. If the server has requested client authentication, the server attempts to authenticate the client (see Client Authentication for details). If the client cannot be authenticated, the session ends. If the client can be successfully authenticated, the server uses its private key to decrypt the pre-master secret, and then performs a series of steps (which the client also performs, starting from the same pre-master secret) to generate the master secret.
7. Both the client and the server use the master secret to generate the session keys, which are symmetric keys used to encrypt and decrypt information exchanged during the SSL session and to verify its integrity (that is, to detect any changes in the data between the time it was sent and the time it is received over the SSL connection).
8. The client sends a message to the server informing it that future messages from the client will be encrypted with the session key. It then sends a separate (encrypted) message indicating that the client portion of the handshake is finished.
9. The server sends a message to the client informing it that future messages from the server will be encrypted with the session key. It then sends a separate (encrypted) message indicating that the server portion of the handshake is finished.
10. The SSL handshake is now complete and the session begins. The client and the server use the session keys to encrypt and decrypt the data they send to each other and to validate its integrity.
11. This is the normal operation condition of the secure channel. At any time, due to internal or external stimulus (either automation or user intervention), either side may renegotiate the connection, in which case, the process repeats itself.

 An SSL-enabled client goes through these steps to authenticate a server's identity:

1. Is today's date within the validity period? The client checks the server certificate's validity period. If the current date and time are outside of that range, the authentication process does not go any further. If the current date and time are within the certificate's validity period, the client goes on to step 2.
2. Is the issuing Certificate Authority (CA) a trusted CA? Each SSL-enabled client maintains a list of trusted CA certificates. This list determines which server certificates the client will accept. If the distinguished name (DN) of the issuing CA matches the DN of a CA on the client's list of trusted CAs, the answer to this question is yes, and the client goes on to step 3. If the issuing CA is not on the list, the server is not authenticated unless the client can verify a certificate chain ending in a CA that is on the list.
3. Does the issuing CA's public key validate the issuer's digital signature? The client uses the public key from the CA's certificate (which it found in its list of trusted CAs in step 2) to validate the CA's digital signature on the server certificate that is being presented. If the information in the server certificate has changed since it was signed by the CA, or if the CA certificate's public key doesn't correspond to the private key that was used by the CA to sign the server certificate, the client does not authenticate the server's identity. If the CA's digital signature can be validated, the client treats the server's certificate as a valid "letter of introduction" from that CA and proceeds. At this point, the client has determined that the server certificate is valid. It is the client's responsibility to take step 4 before it takes step 5.
4. Does the domain name in the server's certificate match the domain name of the server itself? This step confirms that the server is actually located at the same network address that is specified by the domain name in the server certificate. Although step 4 is not technically part of the SSL protocol, it provides the only protection against a form of security attack known as a "Man-in-the-Middle Attack." Clients must perform this step and must refuse to authenticate the server or establish a connection if the domain names do not match. If the server's actual domain name matches the domain name in the server certificate, the client goes on to step 5.
5. The server is authenticated. The client proceeds with the SSL handshake. If the client does not get to step 5 for any reason, the server that is identified by the certificate cannot be authenticated, and the user is warned of the problem and informed that an encrypted and authenticated connection cannot be established.

Also,Found a interesting explanation in Stackoverflow,

First, understand that there are generally two steps in HTTPs communication.

1) Generate a shared symmetric key which can only be known by client and server, no one else knows it

2) With this shared symmetric key, client and server is able to safely communicate with each other without worrying about information being intercepted and decrypted by others.

So the question becomes, how can the client and server generate a secret shared key without being known by others in this open internet? This is the asymmetric algorithm coming to rescue, a demo flow is like below:

-- Client receives public key from server.

-- Client generates a key string "DummySharedKey" which will be later used as shared key, and encrypt it into "7$&^^%####LDD!!@" with server's public key, Man-in-the-middle might be able to intercept the encrypted data, but the data is useless to him as the data can only be decrypted by sever's private key.

-- Server receives the encrypted key string "7$&^^%####LDD!!@", and decrypt it into "DummySharedKey" with its private key

Above key exchange steps makes sure that only Client and Server know the shared key is "DummySharedKey".

So it's critical to understand that it is Client's responsibility to generate the shared key, NOT SERVER! (i think this is what confused you)