In Part 2 we looked at getting some data into NetScaler Insight Center. In this Part 3 we will look at troubleshooting the implementation if you are not seeing any graphs.
First up, as per Part 1 make sure your Firewall ports open if there is one between NetScaler and Insight Center – make sure you are opening UDP port 4739 – not TCP! Another caveat is that the Nitro API traffic is not standard HTTP so if your firewall is doing deep packet inspection, it may need a relaxation rule to not inspect those packets.
The AppFlow traffic will source from the NetScaler IP (NSIP) of the appliance. If you would like to source from a different IP, such as the Subnet IP (SNIP) you can leverage System > Network > Net Profiles and manually specify your collector:
If you do not have a NetScaler (Standard/Enterprise/Platinum) license installed on your NetScaler appliance you will get a message like this in Insight:
Login to the NetScaler UI and check System > AppFlow. Check Collectors is populated with the correct IP of your NetScaler Insight Center VM:
Check Policies has the correct policies for Web/HDX Insight as required:
Check Actions has an AppFlow action that points to the Insight Collector:
Check the AppFlow policies are bound to the correct virtual LB server for Web Insight. First up AppFlow:
Next EdgeSight Monitoring (HTML Injection) if you are using the HTML Injection option:
Finally the Rewrite policy:
Or if using HDX Insight the correct VPN Virtual Server:
Another step is to check that Appflow is enabled on Load Balanced and NetScaler Gateway Virtual servers:
It’s also important to check Load Balanced Services too:
Once those pieces are in place and you have generated traffic you should see some hits for the policies you have defined. System > AppFlow > Policies is a good ‘one-stop shop’ to see all AppFlow policies in one go.
From the CLI you can do all of the above and probably faster!
> show appflow collector 1) Name: af_collector_192.168.47.16 IPv4 address: 192.168.47.16 UDP port: 4739 Netprofile: Done > show appflow action 1) Name: af_action_192.168.47.16 Collectors: af_collector_192.168.47.16 Hits: 284 Action Reference Count: 5 Done > show appflow policy 1) Name: af_policy_LB_CTX_192.168.47.16 Hits: 279 Undef Hits: 0 Active: No 2) Name: af_policy_CS_RGB_192.168.47.16 Hits: 2 Undef Hits: 0 Active: No 3) Name: af_policy_SSLVPN_192.168.47.16 Hits: 3 Undef Hits: 0 Active: Yes Done >
Is HDX Insight working?
Before launching applications or desktops, an easy way to check is whether the Insight Dashboard > HDX Insight > Licenses tab shows any results. Here is a screenshot of my test NetScaler with 10 CCU licenses available:
The next step is to launch an app or Desktop via your NetScaler Gateway and then check in Insight Dashboard > HDX Insight, your NetScaler(s) should be visible in Gateways:
Next up is to check Applications:
Also the Users node populate with data.
Note that the default time-frame for the graph display is 5 minutes – there are options for 5 minutes, Hour, Day and Month. NetScaler licensing was discussed in Part 1 so be careful on which time-frame is supported for your license.
So what happens if you don’t have any data and you’ve checked all the above?
Tracing is likely the final frontier in trying to resolve the issue. This would need to be done on both the NetScaler and the Insight Center VM. We can achieve this a couple of ways, using nstrace or tcpdump on the NetScaler. There’s a technote here on doing this from the NetScaler CLI, or UI here. For the Insight VM, we need to break out the command line again:
# tcpdump udp port 4739 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on 1/1, link-type EN10MB (Ethernet), capture size 96 bytes 14:07:14.887499 IP 192.168.47.1.4038 > 192.168.47.16.4739: UDP, length 45 14:08:14.887650 IP 192.168.47.1.4038 > 192.168.47.16.4739: UDP, length 45
Here we can see some AppFlow records sent from the NSIP of the NetScaler to port 4739 on the NetScaler Insight Center VM.
Upgrading Insight is reasonably easy and should be familiar for those who have upgraded NetScaler. The latest Insight builds are available here and the build file should follow this format build-analytics-10.1-120.13.tgz
Once we have the file we can either use the UI via Configuration > NetScaler Insight Center > Software Images. Select Action > Upload, browse to the file and select Upload. We now have to switch to the command line either via a console or SSH session to Insight Center. Here are the steps below. Change to the /var/mps/mps_images/ directory:
# cd /var/mps/mps_images/ # ls build-analytics-10.1-120.13.tgz
Expand the build file using the tar command:
# tar zxvf build-analytics-10.1-120.13.tgz mps.conf analytics-10.1-120.13.gz analytics-10.1-120.13.sha2 analytics-10.1-120.13-help.tgz analytics-10.1-120.13-mpsds_bins.tgz analytics-10.1-120.13-nitro-java.tgz analytics-10.1-120.13-nitro-csharp.tgz analytics-10.1-120.13-nitro-rest.tgz vmware-tools.tgz analytics-10.1-120.13-gui.tar installanalytics
Run the installanalytics script. You can append the optional -Y switch to give a yes to any prompts during the script
# ./installanalytics installanalytics version (10.1-120.13) kernel (analytics-10.1-120.13.gz) There may be a pause of up to 3 minutes while data is written to the flash. Do not interrupt the installation process once it has begun. Installation will proceed in 5 seconds, CTRL-C to abort Installation is starting ... Copying analytics-10.1-120.13.gz to /flash/analytics-10.1-120.13.gz ... ............................................... Installing analytics GUI... Installing analytics NITRO Java API... Installing analytics NITRO C# API... Installing analytics NITRO Rest API... Installing analytics Online Help... Current DB Version = 9 and New DB Version = 9 Backing up /mps/mps.conf to /var/mps/mps.conf.bak ... Extracting MPS Datastore bins... Rebooting ... shutdown: [pid 12917] *** FINAL System shutdown message from nsroot@ns *** System going down IMMEDIATELY Message from syslogd@ns at Wed Oct 23 10:29:16 2013 ... <auth.emerg> ns init: Rebooting via init mechanism # Connection to insight.nstipster.lab closed by remote host. Connection to insight.nstipster.lab closed.
Wait for the Insight VM to reboot and we should now be upgraded
So there we have it, in this three part series we have looked at all the groundwork needed to implement, populate with data, troubleshoot and upgrade the NetScaler Insight Center product. Enjoy!
Andrew AKA the NetScaler Tipster