SCSI3 PGR: "Want support on Symmetrix? Reboot 500 Windows servers. Continued.."

By Renegade on Monday 31 August 2009 09:27 - Comments are closed
Categories: SAN, Storage, Symmetrix, Windows, Views: 7.677

Alan Shugart introduced something called the "Shugart Associates System Interface" or in short "SASI" in 1981, and created something that can now be called a commodity. He probably didn't realize back then what an impact his new product would have later on.

You can find the SASI, or SCSI as it is now called, standard in a lot of hardware that is being produced in the storage oriented market today. Among others, you will find the standard in disks used in servers, you will find the protocol in fibre channel SAN's and you will find it being used in high availability cluster environments.

The part about the high availability clusters is the part I want to talk to you about today.

I wrote about HA clustering before and one of the parts that is important when it comes to clustering is consistency in the files used in the cluster.

Lucky for us, the protocol designed by Mr. Shugart (in later versions of the standard) implemented something called SCSI reservations. Basically you can send out SCSI commands like for example the 6 byte reserve command. Earlier versions of the SCSI protocol delivered to us something that a lot of people in clusters call "disk fencing", or SCSI-2.

SCSI-2 is based on exclusive reservations, meaning that only one node owns the disk. This also means that the other nodes can't reserve the disk, which can lead to some "undesired" behavior. For example, SCSI-2 is not reboot persistent. Meaning that a node that rebooted and came up, registered the disk and would be allowed read/write access to it. Not the most elegant solution I would say? :+

Now, SCSI-3 PGR works with group reservations, meaning that every node has a key on a dedicated area of the disk and other nodes can simply remove a nodes key to remove the nodes reservation. It also means that a host will need to register after a reboot, and it will have the option of checking the reservation state. This should avoid multiple hosts having read/write access at the same time, if we don't want them too.

Sounds like a useful feature? It is! :)

Now then, back to our problem with the reboot of 500 Windows hosts. After opening a case with EMC, things went a little dormant. Our host base was verified, and as usual we were asked for emcgrabs/emcreports from every attached Windows host in our environment... 8)7

We checked Enginuity versions on our DMX's and the dreaded support matrices from EMC and found that we really did not have an option, except not upgrading and running the risk of falling out of support.

Right now, the situation if even more tense, since Microsoft came out with a new version of the storport driver in a new hotfix. You can find more info on hotfix 950903 here. The problem being that when you run a HEAT report, this hotfix is recommended by Microsoft. But if the FA flags are not set up in a proper manner, you are bound to run in to problems.

Now, here's a small list of currently required flags for the various operating systems:

Windows Server 2003
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2003 with failover clustering
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2008
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2008 with failover clustering
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
  • PER bit for each clustered device (attribute=SCSI3_persist_reserv)
As stated before, these flags are an absolute requirement to get support from EMC, but unfortunately the situation is still more or less the same. It's amazing how slow things can go along sometimes.

All I can recommend right now is to talk to your EMC representative and explain the situation, and ask for a solution. This will affect people more and more, and in my opinion, this needs to be solved.

Again, I will try to post an update as soon as I have newer information that I can share. Until then I for one am keeping my fingers crossed that we don't run in to problems.

It's all about passion and being a geek!

By Renegade on Tuesday 25 August 2009 17:15 - Comments (1)
Category: General, Views: 2.308

I was sick for a couple of days and that got me thinking about my job and the fact that I am a geek.

Yes, I admit it. I'm a geek! Or a nerd if you will. Not in the sense that I'm allergic to sunlight, or have no idea how to talk to women, but much more in the sense that I love my job and like to find solutions to tough problems, or design new landscapes, or even explaining to other departments how the technology in the background works.
Revenge of the nerds
But it seems (enterprise sized) companies don't value that as much as they did before?

I talked to various people from various industries, and to me it seems that a lot of companies tend too sort of "mold" the employee in to something that he is not, or doesn't even want to be.

Outsourcing is a big issue. The market, or more likely the stock holders, demand it. "We will save big bucks, no matter what the cost is!". That automatically means that the role of the person doing his or her job before changes in to something else. You will often hear the term "coordinator" or something like "skill/knowledge holder". And those are the people that worked with a lot of passion before.

But they need to change. These people have a lot of knowledge, but are said to be "too expensive" for their everyday chores. But are they? Are they really?

Or to just paint this picture for myself... I'm in to tech. I enjoy the things I mentioned above, but I am too expensive to actually do those things? I should train people from a different country or even continent and then perform higher level work? I can actually imagine that this might be interesting. But what about others? What about people who are happy were they were, and with what they were doing? Is it actually worth it trying to push people in to different roles, even with the possibility of losing those people?

I am all for evolution. We need to accept new things, and we need to learn if we want to be better. But do we want to be better at the things we are passionate about, or should we become something we are not at the risk of not being passionate about the things we learned?

I know a lot of people with passion for what they do, and you know what? I love working with them, because these people inspire. They are good at what they do, and I would hope that I get the chance to be a little like them in the things I love. But it makes me sad to see them lose their passion. They will get frustrated, or even worse, in the end it won't matter to them anymore. They won't care, and that is probably the worst thing that can happy to a company?

Legends don't die or fade away. Not even the ones about SAP and pagefiles

By Renegade on Friday 7 August 2009 13:43 - Comments (6)
Categories: General, SAP, Views: 3.945

Some things just stay around. You can try to tell people that it's just not true, but they won't believe you, or they will just ignore of forget the things you tell them.

One of the most heard things about SAP and hardware resources is that you need at least twice the amount of RAM configured as swap or as a paging file. Sometimes you will even hear that you need up to three of four times the RAM.

Now, this was actually true.... In 2003 or so. 8)7

Nonetheless I still receive mails about this recommendation. One that came in today was something along these lines:
The SAP installation guide recommendations that the amount of swap space on the server be twice the amount of RAM - this recommendation is valid for Database, Central Instance, and DIA instances.

The following servers all have 32GB of RAM allocated to them. They also all have 32GB of swap configured. In order to comply with the prerequisites we'll need to increase the swap space on each to 64GB as soon as possible.
Now, this would still be ok if we were talking about physical machines that are relatively small. If there wasn't the fact that we also have servers that are quite a bit bigger and carry 256GB of RAM on them. Configuring 2x the amount RAM or even 4x that amount would just be a waste of disk space, and to get this part written down:

I'ts no longer required on IA64 and x64!

People wrote down the numbers of some SAP notes once, and they continue to refer to these same notes instead of just checking if there perhaps is an updated version of it flying around somewhere.

Now this could spark up an entirely different discussion about versioning of documents and the ability to locally store copies of documents. Anyone would have a blast at an ISO 20000 audit. :+

Anyway, to get rid of the rumors in this post, at this point in time you can check the following SAP note #146289 (link may require a valid service marketplace account) which in a nutshell states the following:
When implementing the 64-bit kernel, we recommend at least
20 GByte of swap space. (Plus approx.10 GB for every additional 64-bit instance on the same computer) See note 153641 for more information.
And this is valid for all installations with a 64-bit kernel. If you want to print it, add a date of the last change, or just subscribe to the document, and remember to perhaps check after a year or so if some newer info is available. After all we are working in the IT-business and things do tend to change every now and then. ;)