In my years of both being an employee and consulting I have done plenty of 'making it work' - system administration and deployment- but it seems to me the real value that I've been asked for time and again is the following question:

Are we following Best Practices in our Citrix Environment?

Even the 'Cleanest' Environments Get Dirty Eventually

Sure, a having a quality design is good... for a few years. But the reality is that in the world of application and desktop delivery, last year's practices are already being questioned and turned on their head.

This picture was taken way back in 2003 when I was visiting a Kinko's location in Honolulu. A true testament to "the way they used to make 'em" this server had been trucking along with no big issues... no reason to suspect anything bad could happen. But something bad could have happened!
Now, this is in Hawaii, pretty far away from the main offices and any visiting IT staff. The reality is that noone knew there was even the potential for a problem because the door to the hard drives hadn't been opened in literally years- in fact it was only the fact that I needed to access the CD drive that the panel was unlocked at all. It had just become blended into their overall environment for years and management was unaware of the threat.

Now, of course for my more modern audience of Citrix Workspace (XenApp/XenDesktop/XenMobile) Customers and their Partners very seldom ask me to go to Hawaii (hint, hint) or to even poke into server closets and rooms to look at their physical condition... but the problem in the virtual world is so much worse because the warning signs are so less obvious until it affects nearly everyone.

As a technical leader in your organization, you have two choices here:

  1. Get upset at the world and everyone that sold you a product that needs care and feeding- not accepting things change and you've been left behind.
  2. Adapt to an IT world that is constantly moving forward at the pace of business, and recognize that regular health checks with an expert is something that should be budgeted right along with it to remain competitive.

Knowing Where to Clean is as Important as Knowing How To Clean It

I get it. I have been there. IT staff are stretched to the limits these days- and pretty rarely is there anyone that can be dedicated to knowing everything about a single technology.

I also realize it's hard to take a step back from things and realize what is not going as it should. When most of your day is filled with trouble tickets about resetting passwords and profiles, it's hard to see that user experiences have generally degraded because applications have changed and the system hosting them hasn't.

What you need is the ability to sort thru what is important and what is not, to know where to clean and what can wait.

Let me guide you thru what I do for customers both directly and on behalf of Citrix and their Partners:

Actionable Assessment and Direction- the Infrastructure Assessment

I would argue that the most cost effective and useful use of consulting time these days is the Infrastructure Assessment.

There are a few reasons for this, but the most important is that a quality assessment will detail to management both Tactical and Strategic recommendations in a way that multiple teams and team members can attack the tasks of cleanup, optimization and updated practices... or identify needs beyond what the staff can currently handle.

A quality assessment will be between 5 and 50 pages depending on the level of detail desired and should contain sections that pertain to each area touched by Citrix for Application and Desktop Virtualization, which you can think of as layers of a cake, from bottom up:

  • Business Layer - the foundation and the core 'why' you are virtualizing applications. This is where we want to talk to management and get a feel for not only what is working and what is not, but also lay the foundation for what success should look like overall.
  • User Layer - Identifies the needs and current capabilities of the users, what devices they use and expectations they have. This is where use cases are defined.
  • Access Layer - How the users access their applications and data, often a discussion about Web Interface, StoreFront, NetScaler Gateway and others.
  • Resource Layer - Whether Applications or Desktops- they have to live somewhere and this layer talks about the configuration of the operating system and the applications themselves for each use case.
  • Control Layer - Configurations for the Delivery Controllers and XenApp Zones, Imaging Controllers (MCS/PVS), SQL, Active Directory and policies, etc.
  • Hardware Layer - Physical hardware configurations to host all of the workloads, the underlying hypervisor (contrary to popular belief most Citrix Experts do know a great deal about VMware ESX by the way), the networking and storage components that tie it all together.
  • Operations Layer - One of my favorite yet underserved sections covers who is engineering and maintaining the environment, the way your service desk is running, security procedures and anti-virus, backups, BCDR (a common miss) and overall training of staff and users.

I personally feel that every professional assessment should start with a statement of our understanding of who the customer is and why they are using Citrix, then lead into a high-level (not technical) summary of the current state- good and bad, and then list the top Short and Long Term steps to be taken. Some customers would like a score of each layer in regards to the level of risk in each.

Tactical Course Correction - The Health Check

Health Checks are often overlooked or mistaken for full assessments. Health Checks typically involve a rapid dump of data and checkups from the actionable items defined in the last assessment.

I will often do Health Checks free to customers on behalf of partners that would like to know how to best direct their efforts and any points that require follow ups. The typical Health Check will produce a 1-4 page memo outlining high-level scores for limited areas within the layer system I noted above.

For example, I spent an afternoon with an IT manager last year and we identified ten areas in which their environment was either following or not following the most common leading practices. I have expanded this slightly and now refer to this as the 'dirty dozen' or the 12 things I see most commonly on assessments and how to follow up with them.

Health Checks are NOT good for avoiding risk in the absence of an Infrastructure Assessment, but they are valuable as a follow up or helping to determine next steps for the quarter or half of the year.


 

Quality Recommendations are Critical

I'll go ahead and say it:
I've been seeing some pretty bad advice being given out these days. It is pretty easy to have happen, really. There is enough opinions out there - blog posts, changes to documentation and just plain old fashioned FUD to seriously muddy the waters.

There's a few things I'll advise here:

  1. Experience Matters. I have been doing Citrix work as a focus for well over 12 years now - but I only started doing professional assessments about 4 or 5 years ago.  The reason is that even though I'd been making it work for a long time, it takes more than that to really give quality opinions. My assessment skills were honed by none other than the architects at Citrix Consulting, leading me to my next point.
  2. Accountability Matters. What I loved about Citrix Consulting and why I still do work with them on occasion is that they do have an interesting approach to things like assessments. Usually a somewhat seasoned consultant (2-3 years strong typically) will lead the engagement with a more junior consultant learning the ropes... but their opinions aren't the final say. They submit their findings to an Architect with 5 or more years of experience who looks at the whole picture and spots things that others may miss... but more importantly they look with eyes that see what is working and not working in the industry- to be able to identify leading practices and what lowers risks. That second and third set of eyes is important and even when I am on my own for an assessment, I'll very often have another trusted expert look at what I'm saying and check my opinions against what they have seen. This dropping of pride and inserting accountability is very valuable in my mind.
  3. Knowledge Outside of Citrix Matters. Citrix technologies depend on many others to work. If your assessment doesn't include a deep look at your Active Directory, SQL, Networking and Storage infrastructures at a minimum (or you deny the consultant access to these areas) you simply will not get a good understanding of what is going on and how to best clean and tune your Citrix environment. Over the years and tweaks to it, I still agree with the way Citrix Consulting structures their assessments - in Layers to cover all of these concern areas, not just the Citrix technologies... which brings me to the next painful point.
  4. Sales Engineers do not good Consultants make. Don't get me wrong- I love the relationships I have with the Sales Engineers that I know. But honestly- when it comes to your environment's health, it is best to take the advice of consultants that are not tied to sales. SEs have a lot of things to balance, and are not always trained on the nuances of leading practices. Their training tends to be focused towards product features and leading POCs, which is good. But eventually someone needs to get focused on what features will really work in your unique situation that can write them down professionally in backed recommendations... See #1. Love you SEs out there, but seriously- stop trying to play this role. Your job is to guide purchasing decisions, not configuration decisions. If you don't know the difference you may damage your customer and therefore your relationship. To everyone else I'll simply say that if an SE is telling you how to configure things in production without regard to testing first- kick them out of your building. And for those of you who want to take your POC into production... well, I'll probably see you in a year to fix it with an expensive re-design so I suppose go right ahead.
  5. Choose a professional consultant that cares about your business. If you have someone come in with their own agenda, own thoughts or worse- no desire to learn about what your business cares about - you won't get what you pay for, even if it is a bargain. This is one of the things I harp on with consultants I mentor- if you don't understand the use case and the needs of the business, you are just another admin or engineer that is better suited to being told what to do rather than advising a CIO as to what should be done. It is actually something that irritates me greatly in the IT field today, and I want to make that distinction. Frankly, if "Tell me in your words what your company does and what they care about" aren't pretty much the first questions that are asked, you probably need to get a new 'expert'.
  6. Impressive Titles do NOT matter. I will probably catch some flak for this, but just because I have a CCE-V certification does not say I know what is best for your environment. Because I have a job title of Architect does not make me know what is best for your environment. What really matters is what I mention above. While certifications and such are important - there is currently no certification for third-party assessments. Until that day comes- you need to do some research and find out if the person conducting your assessment can be trusted to guide you well.
  7. Impressive Tools do NOT matter. I have been getting a little frustrated by this lately. Some tools are very useful for making the tasks of documentation easier, but what I am finding is that while pages upon pages of information may seem impressive - an assessment is just that: tell the customer what is wrong and why. Tools that grab configuration information and run checks can tell you a lot about the health of the configured environment- but can't tell you anything about the WHY it was configured. If you want an assessment with me, it's best to prepare your team for some long 8 hour days of talking to me. I'm going to want to know the whole big picture, because that is what you're paying me for! Anyone can run assessment tools - most of them are free or fairly cheap. Find out in advance if your consultant intends running a bunch of assessment tools but has no plans to talk to the various IT teams for each focus area... If you just want a current-state configuration document, that's great. Otherwise you need to assess your assessor more carefully!
  8. TRUTH ABOVE ALL. This is the motto I have for my business. What you need to hear may sometimes be hard to hear, and the assessment is the time you'll be facing those uncomfortable things... but you need to find someone willing to tell you what you need to hear. If the person conducting your assessment is suggesting products or services without any real justifications for example - they may not be giving you all the information you need. But if the consultant is telling you to consider products A, B and C because you have a gap in your security... perhaps not dismissing it as 'sales talk' would be wise. Look for someone who takes truth seriously- "Insulting Consulting" as my friend Jake Hughes calls it :-)
    The bottom line here is that even if it is hard to hear, in my experience hearing the truth always turns out for the best.
    I tell anyone I work for that I will not lie or misrepresent anything. It has honestly cost me a few projects and some money... but I sleep better at night knowing I did right by my customers, because that is more important.

 

Get Clean... Stay Clean!

So now that you know how I feel about the matter- 

  1. Take this seriously. I've said my pet peeve about consultants, now to my pet peeve about management. Infrastructure Assessments and Health Checks are valuable tools to help you meet the needs of your users... but only if you do something with them. It is easy to get bogged down with other tasks, and even to hand the document off to have someone else read... but a quality assessment will have a first page that is written as an Executive Summary. Read it. Take it seriously and ask questions. The rest of the document will have things to delegate but at a minimum it is important to take the time to get the consultant's view of what is going on and either agree or disagree.
  2. Schedule Regular Checkups. Like taking your car in for maintenance or going for a yearly physical or dental checkup- if you want to keep things clean and running well, being proactive is so much better than being reactive. I have seen FAR too many times customers that rejected the notion of an assessment make multi-million dollar mistakes that could have been prevented with a proactive assessment. The case I'm thinking of actually made headlines and when the team was brought in to do a panicked assessment the cause was simple and would have been prevented had they not relied only on their own IT team to have eyes on the configuration.
    Make assessments part of your yearly budget. You may even want to challenge your team to define KPIs based on the assessment, so that the next assessment shows improvement.
    As far as the regularity, I typically recommend a 4-2-1 schedule: 4 the first year (1 week+ assessment, then 3 health checks), 2 the second year (1 assessment, 1 health check) and then 1 assessment each following year.
  3. Don't Go Cheap. A one week assessment from a reputable source should cost between $8,000 and $25,000 depending on the level of detail and staff involved. A 1 to 2 day Health Check should be about $1,500 to $3,500 in most cases. To be blunt, anything less and you've probably attracted people that aren't qualified and may not get truly actionable steps as a result. I hate to be in the position of saying this, but in all honesty- do you really want to trust your environment to the lowest bidder rather than the most quality bidder?
  4. Look for a Relationship after the Assessment. Now the one thing I don't think I like quite as much about consulting. While it is valuable that I am able to go all over the country and see first hand what is working and what is not... I don't always get to build relationships with my clients, which is sad. So my advice is to get involved with a partner that cares about consulting as much or more than they care about sales... have them provide assessments from members of their Professional Services team or ask them to either work with or outsource to a trusted third party like myself. Now I know, that sounds like a very odd thing to say but what I tell my customers is that the things I recommend typically need efforts and product needs beyond what they have internally. Having a trusted long-term relationship is important. I can point you in the direction of some partners I trust, and when I am conducting independent assessments, quite often do.
  5. Follow-On Work. I'll also mention here that many experts that do assessments can also help you with the action items. If you need a new design, for example- most of the legwork has been done - you'll save time and money by engaging the same team to take the next steps with you!

So there you have it- just a few of my thoughts on assessments and health checks... I have many more, but these are the big keys.

Do you agree with my list? I'd love to hear from you!

Now the fun part! As of the day I'm writing this, we have five 1-week assessment slots to fill starting in mid-March, but I'm hoping to get more like TEN by the end of May. As a celebration of Spring Cleaning and simply to try and get more of these going in general here's what we are doing:

The first five NEW customers to sign up for a 1-week assessment my May 30th will get $1,000 off of the first assessment, and $250 off their first health check if completed by August 31st. Contact me directly via LinkedIn or by going to http://bot-consulting.com and click on "Contact Us".

The first five NEW Partners to sign up for a 1 week assessment for their customers by May 30th will get $500 off and $200 off their first health check if completed by August 31st. Please contact me by either LinkedIn or by going to http://eshelmanenterprises.com and clicking... you guessed it- "Contact Us"

But wait...  if you REFER a new client to me that completes an assessment- you'll get your choice of $200 service credit, or a $100 gift card. This offer expires May 30th, so I'd love to hear from you as soon as possible!

 

Thanks for taking the time to read this! Comments are welcome!

 

(this article was written for LinkedIn at https://www.linkedin.com/pulse/spring-citrix-cleaning-season-dj-eshelman but is posted here as well because... well, I wrote it and I think it's valuable)