Category Archives: Netscaler

  • 2

A Citrix Active/Active Deployment Part 3

I’m at the point now where the Datacenter#2 has Networking, Storage, and Hosts ready for me to use.  That means it’s VM building time.  You pretty much need the identical components you have for your existing site.  Remember the key focus here is Highly Available and reliability.  So if one datacenter goes down, the other one will be able to take over.  This means we need the same VMs with the same roles.

  • 2x – Storefront Servers
  • 2x – Delivery Controllers
  • 2x – Director Servers
  • 2x – SQL ‘Always On’ Servers
  • 2x – PVS Servers
  • 1x – Licensing Server

Licensing: Usually in Active/Active scenarios you would deploy a Licensing VM and allocate/purchase the same amount of Citrix licenses as the first site.  Since this wasn’t budgeted we will be using a manual failover type of approach when it comes to the licensing server. We’ll be sending both Datacenters to one licensing server.  If that site fails or the primary licensing VM fails, we will manually switch the Citrix Sites to use the backup licensing server.  So technically we will be building a VM, installing the Citrix license role, but not allocating any licenses to it.  This should pass any ‘audit’.  Note:  You technically have 30 days to get the license server up before functionality goes away.  That should be enough time to backup/restore, build new, or whatever.  This backup licensing server will most likely be used in an ‘Ohh SH!…’ moment.

Here is a blurb about ‘Grace Periods’ from Citrix docs – https://docs.citrix.com/en-us/licensing/11-14/technical-overview.html

Grace periods

If product servers lose communication with the License Server, the users and the products are protected by a grace period. The grace period allows the product servers to continue operations as if they were still in communication with the License Server. After the Citrix product checks out a startup license, the product and the License Server exchange “heartbeat” messages every five minutes. The heartbeat indicates to each that they are still up and running. If the product and the License Server don’t send or receive heartbeats, the product lapses into the licensing grace period and licenses itself through cached information.

Citrix sets the grace period. It is typically 30 days but can vary depending upon the product. The Windows Event Log, and other in-product messages, indicate if the product has entered the grace period, the number of hours remaining in the grace period. If the grace period runs out, the product stops accepting connections. After communication is re-established between the product and the License Server, the grace period is reset.

The grace period takes place only if the product has successfully communicated with the License Server at least once.

Grace period example – two sites, both using the same License Server

The connection between Site 1 and the License Server goes down causing Site 1 to go into the grace period, continuing operation and making connections. For concurrent licenses, they can connect up to the maximum concurrent licenses installed. For user/device licenses, they have unlimited connections. When Site 1 reestablishes communication with the License Server, connections are reconciled and no new connections are allowed until they are within normal license limits. Site2 is unaffected and operates as normal.

If the License Server goes down, both sites go into the grace period. Each site allows up to the maximum number of licenses installed. As above, the user/device licenses have no limit.

Storefront:  There are some key considerations when building out the storefront servers with multi-site design in mind.  Please read ‘StoreFront high availability and multi-site configuration’  – https://docs.citrix.com/en-us/storefront/3/sf-plan/sf-plan-ha.html.   Once we get everything setup, I’ll expand upon aggregating sites.  Stay tuned for a future blog post on those specifics.

When you decide whether to set up highly available multi-site configurations for your stores, consider the following requirements and restrictions.

  • Desktops and applications must have the same name and path on each server to be aggregated. In addition, the properties of aggregated resources, such as names and icons, must be the same. If this is not the case, users could see the properties of their resources change when Citrix Receiver enumerates the available resources.
  • Assigned desktops, both pre-assigned and assigned-on-first-use, should not be aggregated. Ensure that Delivery Groups providing such desktops do not have the same name and path in sites that you configure for aggregation.
  • App Controller applications cannot be aggregated.
  • Primary deployments in the same equivalent deployment set must be identical. StoreFront only enumerates and displays to users the resources from the first available primary deployment in a set, since it is assumed that each deployment provides exactly the same resources. Configure separate equivalent deployment sets for deployments that differ even slightly in the resources they provide.
  • If you configure synchronization of users’ application subscriptions between stores on separate StoreFront deployments, the stores must have the same name in each server group. In addition, both server groups must reside within the Active Directory domain containing your users’ accounts or within a domain that has a trust relationship with the user accounts domain.
  • StoreFront only provides access to backup deployments for disaster recovery when all the primary sites in the equivalent deployment set are unavailable. If a backup deployment is shared between multiple equivalent deployment sets, all the primary sites in each of the sets must be unavailable before users can access the disaster recovery resources.

What I did is built my new VMs, installed Storefront, and joined them to my existing ‘Storefront group’.  This way, everything would have the same names, setup, and configurations.  Afterwards, I disjoined the ‘group’ and added only the storefront servers at Datacenter#2 to a ‘storefront group’.

Director Servers:  For director you will need to follow the Multi-Site instructions – https://support.citrix.com/article/CTX136165 . The overall goal for this will be to have Director at each Datacenter be configured for ‘Multi-Site configuration’.  We will use GSLB to send users to a LB VIP at either Datacenter.

Provisioning Services: We will treat PVS as separate entities.  So build out PVS like you would normally from scratch.  Since we use Unidesk/App Layering we will setup a 2nd ‘app layering connector’ that points to Datacenter#2 PVS servers.  Now, I don’t literally have to publish the template at both locations.  ‘Publishing’ compiles all the different layers, which takes time.  So we could setup a powershell sync/transfer script to copy the VHD to the second PVS store.

SQL ‘Always On’:  Since we will be deploying a Site at each Datacenter, technically that site should have it’s own Servers.   Deploy how you would normally deploy in a standard 1 site configuration.

Delivery Controllers: At this point just build your Site as you normally would.  In another post I’ll go through our different options on how both sites will talk to Storefront, and which sets of icons will be available based on settings.  Initially, I’ll be using my homeboy’s, @ryan_c_butler, replication script to export my current Citrix sites information to Datacenter#2 Citrix site.  This is something that can also be used as a scheduled task to keep both environments in sync.

Over the next week or so i’ll be verifying normal functionality of the site, storefront, netscaler, etc…  After that, i’ll be getting into the Storeront and GSLB configurations.

Stay tuned….

Previous Posts:

Part1 – Introduction – http://verticalagetechnologies.com/index.php/2017/04/11/a-citrix-activeactive-deployment-introduction

Part 2 – http://verticalagetechnologies.com/index.php/2017/06/06/a-citrix-activeactive-deployment-part-2


  • 2

A Citrix Active/Active Deployment Introduction

 

Ok, where does one begin when talking about an Active/Active Citrix Deployment?  I’m frazzled just thinking about it.  Do I start with the Profiles, Netscalers, PVS Images, XenApp Sites…?  Let’s first describe the requirements.  Management has instructed the team to build a Highly Available and Stable environment that spans multiple data centers. Ready…Go!

However vague that is, we will let best practice, use case, and experience dictate the rest.

Our Data Centers are located within 10 miles of each other with dark fiber connecting the two together.  From a response standpoint we should have 50-100Gbps of bandwidth with less than 2ms latency.  Hardware wise, each Data Center will have their own Networking, Storage, Hosts, Internet, Netscalers, etc…

Here are some of he major topics I’ll be covering in more detail in this series of blog posts.  I’m not sure if this will be a 4 series blog post or a 10 series blog post.  I’ll keep writing more and more if the community wants to see more.

Netscalers/Storefront:  High level SDX/VPX placements and configuration for Active/Active setup utilizing GSLB.  What load balancing rules will I follow? How will DNS be used for Internal and External users?

Storage and Hosts: Placement of NAS.  Do files get synced to both locations?  How should my documents files be handled?

XA/XD Sites:  Should I have one or two sites?  Should users have a primary data center?  What will Active Directory group structure look like?

Provisioning Services: Will I have separate PVS deployments, or should they be under the same site?  How should images be synced to the 2nd location?

Profiles: How will profiles be synced across data centers?  If someone logs into Site A in the morning and then logs into Site B in the afternoon, will they have the same settings?

While we may not have all the answers at the moment, here are some soft directions that we have been given as well.  We need to be able to implement changes (host patching, PVS version updates, etc…) at the site level, fail over the site that the changes are taking place in, test those changes, then put users back onto said site, and perform the changes to the other site.  I realize this could all be accomplished in a target format from within a site, however, this is the direction we have been given.

Now, there is not ‘right’ way to accomplish this, especially with the different variables of where the data center is located, the storage back end, and the customer requirements/directives.  As I’ve said before, there are 100 ways to skin a cat.

In the following blogs i’ll get more into the details surrounding the technical setup of the primary components to complete our Active/Active setup.

_____________

Part 1 (this blog)

Part 2 


  • 1

Citrix – Smart Access – Conditional Clipboard Access

SmartAccess and SmartControl let you change ICA connection behavior (e.g. disable client device mappings) based on how users connect. Decisions are based on NetScaler Gateway Virtual Server name, Session Policy name, and Endpoint Analysis scan success or failure.

Carl Stalhood has a great blog on how to configure/setup Smart Access.  You can find it here: http://www.carlstalhood.com/smartaccess-smartcontrol-netscaler-11/

Smart Access uses Netscaler Universal Licenses.  Platinum Netscaler Licenses includes this feature, but Enterprise and Standard editions required you to pay for however many you want (usually based upon concurrent users).  This is still the case today.  However, Citrix has recently changed the quantity of Universal Licenses they provide you out of the box. As of Netscaler 11.1.49, New CCU packaging and pricing have changed in the following ways: https://docs.citrix.com/en-us/netscaler/11-1/about-the-netscaler-11-1-release/whats-new-in-previous-11-1-builds.html

1. MaxAAA is automatically set to the maximum licensed number.

2. Licenses now use the following scheme:

a. Platinum: Unlimited (formerly 100)

b. Enterprise: 1000 (formerly 5)

c. Standard: 500 (formerly 5)

d. Any other license: 5

e. If additional CCU licenses are present on the system, you add those to the above values

(for example, standard is 500 + any additional CCU licenses).

f. Disregard additional CCUs for the platinum case, since platinum is already unlimited.

This is awesome news for the community as we’ll be able to provide this functionality at no additional cost, for most organizations.

Smart Access is such a powerful tool and there are many use cases that could be used:

  • Control Device Drive mappings for certain Netscaler Gateways
  • Control Visual Settings/Experience for External/Internal Users
  • Control Application/Desktop icon visibility
  • Etc…

As you can see you can pretty much control and configure any Citrix Policy depending on different access conditions.  This is handled by ‘tagging’ the session as it comes through the Netscaler Gateway via some sort of ‘Netscaler session policy’ condition.  Once the session is ‘tagged’, you’ll create a Citrix Policy with some setting defined filtered by ‘Access Control’.  You’ll define the Netscaler vServer name and the ‘Smart Access’ policy name in the ‘Access Control’ filter in Studio.  When you’re done, you will have a citrix policy is being applied/not applied based on your Netscaler session policy parameters.

In this blog, I’ll focus on controlling Clipboard Access for Internal/External connections.  Here are my requirements.  (Note: Instead of Clipboard, you can substitute that for any configurable Citrix Policy)

  1. Disable Clipboard Access for anyone connecting to Citrix from an External/Untrusted location
  2. Enable Clipboard Access for anyone connecting to Citrix Internally (By default clipboard redirection is allowed)

I’ve Outlined 4 different stages to get this completed:

  1. Prerequisites
  2. Netscaler Configuration
  3. Citrix Policy Configuration
  4. Testing

Prerequisites:

Follow the rules for setting up Smart Access using – http://www.carlstalhood.com/smartaccess-smartcontrol-netscaler-11/

Note: If you are using a Content Switch to direct users to Netscaler Gateway (with a non addressable IP), you might have to adjust the Content Switch Policy or create a dummy ‘Call back’ vServer to address the Storefront Callback setup.

Netscaler Configuration:

  • Create a Netscaler Gateway Session Profile (Don’t configure any settings, just a blank Profile)
  • Create a Netscaler Gateway Session Policy.  Use the blank profile configured above.
    • Expression: REQ.IP.SOURCEIP != 192.0.0.0 -netmask 255.0.0.0
      • Feel free to addon with ‘&&’ and/or ‘||’ operators

As you can see I’m looking to see if the user’s endpoint IP Address in NOT in the 192.x.x.x range.

Note: The Session Policy is configurable based upon any Expression you’d like.

  • Bind the newly created Netscaler Session Policy to your Netscaler Gateway vServer
    • Note:  This needs to have a lower priority over all other session policies, so it get’s evaluated every time.

Citrix Policy Configuration:

  • Setup the Smart Access Control Filter in a Citrix Policy (Studio)
    • Create a new policy with ‘Clipboard Redirection’ disabled.
    • Configure ‘Access Control’ as the filter, click Assign and enter in the relevant Netscaler Settings
      • Farm name: is the ‘FarmName:’ value from director (in the director screenshot below)
      • Access Condition: is the name of the Netscaler Session Policy (highlighted in the screenshot above)

Testing Smart Access

  • Confirm the expression is working from Director
    • As you can see, under the SmartAccess Filters tab, you can see the session policy is added.
      • This is because I connected externally, so my IP address IS NOT 192.x.x.x.
      • Note: The Source IP is the Endpoint IP, unless you are routed via NAT.  When you come through a NAT.  Your Source IP becomes your Public IP.
        • This is why the Netscaler Session Policy is true, because i came over an external connection, via NAT.  Thus NOT having an IP address in the 192.x.x.x range, thus making the policy True, thus getting added (or ‘Tagged’, as i like to say) to the SmartAccess Filter.

Well, that should just about do it.  You’re end result is that you won’t be able to copy/paste (clipboard redirection) for users connecting externally.

I’d love to see this process made a little bit more seamless.  Switching the vServer to ICA Mode and being dependent on Universal Licenses should go away IMO and be available Out of the box. Citrix’s Universal License change in Netscaler 11.1.49 is a step in the right direction though.

Thanks to Carl Stalhood and Beau Smithback for resource documentation and help with Setup/Configuration.


  • 7

Netscaler-11-Disclaimer-Footer-on-Gateway

Netscaler-top-logon

Many companies wish to customize portions or add their little quirks/branding to sections of the Netscaler Gateway/Unified Gateway logon page.  With the introduction of 11.x, customizing the logon page has became increasingly easy.

 

Netscaler-Disclaimer-EULAThis blog post will cover adding a disclaimer/footer to the logon page.  This can be done via the ‘EULA’ section of the NG.  This approach adds a ‘checkbox’ that you must check before you can continue.  However, you don’t actually need to read any part of the EULA/Disclaimer.   You can simply just check the box and continue on your way.  This might not be suitable for some companies as the user doesn’t need to actually read the text to continue.

You can use ‘Rewrite’ Policies to add a disclaimer/EULA/Footer to the logon page that is 100% shown all the time.  This Rewrite policy can be bound to any Netscaler Gateway vServer.  I’ll give you the commands to create the Rewrite policies.  This method is probably the easiest as the GUI is not intuitive enough to perform this without in-depth knowledge.  Not to mention the information/code seems to change for each version (10/10.1/10.5/11).  It would be great if Citrix documented this more thoroughly, or provided us a simple web gui that denotes each section of the logon page that corresponds to the Rewrite policy pattern.  As a bonus I added a URL to the bottom as well. (You’ll notice that I insert “+” every so often.  This is to get around the 255 character limit.)

Here are the 2 commands that you run:

add rewrite action rw_act_insert_loginfooter insert_before_all “HTTP.RES.BODY(120000).SET_TEXT_MODE(IGNORECASE)” q{ “var login_footer=$(\”<div style=’text-align:center;font-size:15px;color:white;’><br>US Government Notice and Consent. AUTHORIZED USE ONLY. <br><br>You are accessing a COMPANY system which provides access to a U.S. Government information system, which “+” includes: (1) this computer, (2) this computer network, (3) all computers connected to this network, and (4) all devices and storage media attached to this network or to a computer on this network. This information system is provided for U.S. “+” Government-authorized use only. Unauthorized or improper use of this system may result in disciplinary action, as well as civil and criminal penalties.<br><br>”+”By using this information system, you understand and consent to the following: You have no reasonable expectation of privacy regarding any communication or data transiting or stored on this information system. At any time, and for any lawful “+” Government purpose, the Government may monitor, intercept, and search and seize any communication or data transiting or stored on this information system. “+” Any communication or data transiting or stored on this information system may be disclosed or used for any lawful Government “+” purpose.<br><br><a style=’color:yellow;font-size:15px’ href=’http://verticalagetechnologies.com/’>Forgot Password</a></div>\”).appendTo(logonbox_container);”} -pattern “box_view.prepare_view();”

add rewrite policy rw_pol_insert_loginfooter “HTTP.REQ.URL.CONTAINS(\”gateway_login_view.js\”)” rw_act_insert_loginfooter

Next, you’ll go to the NG vServer in the GUI and add a binding for the Rewrite policy. Netscaler-Rewrite-Bind

Tip:  When testing to confirm everything worked, you’ll need to clear your browsers’ internet cache to see the changes right away.

Result:

Netscaler-Disclaimer-EndResult

That’s it.  You now have a working disclaimer.

For those doing two factor authentication, I haven’t found a way to change the 2nd password field using a Rewrite policy on 11.x.  For now you are better off creating a custom theme.

Source: http://discussions.citrix.com/topic/371942-netscaler-11-theme-customization-how-to-add-links-and-verbiage/


  • 4

Netscaler Performance Woes…

Hey all and welcome to my first blog post.

I was recently involved in a new Netscaler implementation. Two Netscalers (5550 MPX) were purchased and will be configured in a HA pair.  They currently  have 2 other Netscalers (5500 MPX) that are used for production.  This project is to setup the new Netscalers in the new DMZ and migrate services.

The client chose a 3 arm approach due to a newly archicted DMZ setup.  1 arm goes to 1 interface, this will be dedicated to management traffic.  The other two arms will be configured through 2 interfaces setup in a LACP bond and tagging two vlans.  After a few days some routing problems a colleague of mine shared with me a nice feature for multi-arm configurations, called Mac Based Forwarding.  This essentially communicates to services going through the same gateway/interface it came in on.  This helps with hairpinning, which is when the router/security device will drop traffic because it expects it to be coming from a certain gateway, but it arrives on another.

Setting up the device was pretty straightforward, especially now with the wizards.  I initially started on 10.5 and then installed 11.x shortly after it was released.  I’m at the current build at the moment, which is 11.0.63.16.  Shortly after upgrading the 11.x release i noticed some performance problems.  I really didn’t even notice them until a couple weeks after I upgraded.  The ‘performance problems’ were hit or miss, but basically the symptoms were unresponsiveness, laggy user interaction, and slow repaint issues with moving windows on the screen.  For instance, the screenshot below is a picture of the netscaler GUI login window.  This would take 5 seconds to repaint itself, loading some of the blue dots a section at a time.

Netscaler-Login

I started comparing the performance with the other Netscalers that were already setup.  I could connect to the same Server 2008R2 XA65 session server and the experience was flawless.  Connections straight to the session server were comparable to their currently setup netscalers.  This left me with a couple isolation steps.

Straight to the server was good, and the old netscalers are good.  So the new netscalers are bad….what’s the difference?

The new Netscalers connect to a different DMZ, which connects to the same ASA, but separate DMZ cisco switches, which does more port permit/deny rules compared to the old Netscalers, which do a straight NAT to the internal network.

Throughout the troubleshooting process I tried:

  • Creating new AG VIPs
  • Adjusting different TCP profiles
  • Adjusting Cache Parameters
  • Downgrading to 10.5 and 10.1
  • Disabling TLS on backend services (https://www.citrix.com/downloads/netscaler-adc/firmware/release-110-build-6316.html)
  • Testing traffic through the management port
  • Breaking the LACP bond and testing with 1 interface
  • Adding an arm into the internal network, bypassing the DMZ/firewall completely
  • Removing Features/modes
    • Integrated Caching
    • HTTP Optimization
    • Front End Optimization
    • Appflow
    • Session Reliability
    • Basically everything, but the bare minimum.

I soon correlated that performance problem to the ‘available bandwidth’ metric using HDX Monitor.  When I experienced the lag, I saw the ‘available bandwidth’ metric drop significantly.  The RTT and ICA RTT would skyrocket, and ‘badwidth used’ would go up to 100%.

HDX-Mon-bad

I confirmed this with perfmon statistics along with Netscaler Insight Center stats.

I could simultaneously connect to the same backend server on the old Netscalers using a different username, using the same endpoint/internet connection.  I’d get these results.

HDX-Mon-decent

I also did a couple Bandwidth/Speed tests to the location where the datecenter is located. http://www.speedtest.net/ .  Everything coming back normal.

I did notice there were other people experiencing similar situations, with no actual fix, but to restore to a backup or an earlier version. https://discussions.citrix.com/topic/370744-slow-performance-going-through-ica-proxy-connection/ 

After opening a support case and some inital data gathering, we started capturing Traces on the Netscaler and Wireshark packets on the endpoint/Session Server.  One thing that might save you some time is using the new ‘Decrypt SSL’ featuer on the Netcaler trace.  This essentially decrypts the SSL packets so Citrix can inspect them easier.

Netscaler-ssl-decrypt

After they received the latest capture/trace file the ticket was escalated to level 3.  They were able to confirm this is a known bug, WITH A WORKAROUND.

Hello Steve,

The performance degradation is due to a regression introduced in 10.5.59.x builds which is being currently tracked under BUG TSK0606493.

Workaround/Fix:

The workaround for now would be to change the tcp flavor to BIC from default by executing the below command.

set ns tcpProfile nstcp_default_XA_XD_profile -flavor BIC

Also, enable SACK on the TCP Profile for XA/XD, we will go over this shortly on the GTM Session.

You could imagine my excitement when I received this email. A month of troubleshooting and tinkering only to be resolved by a couple steps.  I applied the changes and confirm by AG VIP was using the ‘nstcp_default_XA_XD_profile’ TCP profile.  I didn’t notice the change in performance until I disconnected/reconnected.

Here are my new HDX Monitor results.  I haven’t experienced any fluctuation in performance since applying the changes.

HDX-Mon-good

So far so good on my performance issues!  Support did say this will be fixed in the next version. I hope I save you all some time.

**********************************************************************************************************************************

**********************************************************************************************************************************

**Update** (12/9/2015)

About 5 hours later I noticed the ‘Available Bandwidth’ metric starting to slowly creep down.  It finally stabilized around 700Kbps.  The session does still seem responsive, a lot more than before.  However, this metric should not get this low.  Again, I connected up to the old Netscalers simultaneously and confirm the 10+Mbps metric.

I visited a couple sites to confirm the exact checkboxes needed for the XA/XD TCP Profile and I ended up with this result (I confirmed this with Citrix Support):

Netscaler-TCP_profile

After making the change I disconnected/reconnected my citrix session.  I’m again seeing the expected ‘Available Bandwidth’ metric.  I’ll give this a couple days to confirm stability.

************************************************************************************************************************************

************************************************************************************************************************************

While the ‘workaround’ i was provided doesn’t seem to 100% fix the issue, the results are still so much better than doing nothing, other than reverting to a previous version.

************************************************************************************************************************************

************************************************************************************************************************************

**Update** (12/14/2015)

After a few days I noticed the ‘available bandwidth’ was higher, and it only dropped to around 800Kbps.  I wanted to 100% confirm a rollback would fix the issue.  I ended up downgrading both Netscalers to the 10.5 safe harbor build (10.5.56.22).  So far so good with connection stability.

Citrix Support said this will be permanently fixed in version 11.0.65.x.  They did mention that 11.0.64.x will be released in a couple days, but will not contain the fix.

**Update** (3/23/2016)

Citrix just released Netscaler firmware version 11.0.65.32. With this release the bug is listed as fixed

With the default TCP congestion control, a NetScaler appliance recovering from packet loss reduces the congestion window to half its previous length. With multiple packet loss events, the congestion window becomes small and delays transactions.
[# 606493, 601655, 623185]
I haven’t had the chance to test this yet. Update this post when I have some final confirmation.

Twitter