Symmetrix access control: When unique is everything but unique

By Renegade on Friday 2 October 2009 10:22 - Comments (2)
Categories: SAN, Storage, Symmetrix, Views: 4.549

So, youve got a million dollar storage box standing there and want to make sure that it's secure? Sure thing you want to do that! And you ask your vendor "What can I do?". One of the replies could to use access control lists or ACL's. And all is great. Or is it?

From what I have heard, very few EMC customers in Europe tend to use ACL's on their Symmetrixes. Perhaps even for a good reason?

If you take a look at the documentation on Powerlink you can find some technical papers on Symmetrix Access Control, and the papers will state (among others) the following:
Today, anyone with access to Symmetrix-based management software can execute any function on any Symmetrix device. Many product applications such as EMC® ControlCenterTM, TimeFinder®, SRDF®, Optimizer®, Resource View, Database Tuner, and various ISV products can issue management commands to any device in a Symmetrix® complex. Open systems hosts can manipulate mainframe devices, Windows hosts can manipulate UNIX data, and vice versa.

Shared systems, such as these, may be vulnerable to one host, accidently or intentionally, tampering with another’s devices. To prevent this, the symacl command can be used by an administrator of the Symmetrix storage site to set up and restrict host access to defined sets of devices (access pools) across the various Symmetrix arrays.
Now, I have to admit that this info is from an older version of this guide, but the same is still true for the most part. You can change to in-band or out-of-band management, you can use the Symmetrix Management Console, but as soon as you install Solutions Enabler on a client connected to the storage box, you more or less open up a world of possibilities on said client.

Usually you don't want that, so why not implement some restrictions? symacl is just the thing for that! Normally I would create an access pool, in which I define permission to a host to perform certain Solutions Enabler functionality or commands on a specified set of devices. These sets of devices are referred to as access pools.

Now, once I have set up these access pools, I can assign single clients or groups of clients to these pools. I do that by creating access control groups. These contain unique access IDs and names, and are assigned to hosts and sorted into access control groups

So now I have one (or more) clients that I allow a certain piece of functionality or a certain (set of) command(s). In order to uniquely identify my client, I can run the following solution enabler command:

symacl -unique

and will receive an output similar to this:

The unique id for this host is: 254A30A9-54319DC0-8A476069

Now that we have the unique host id, we can add id to the configured access group via a command file using the normal preview, prepare and commit routine. After that, you should be good to go.

And that is where things can get nasty.

As we have found out the hard way, a unique host id is not necessarily unique. We have had occasions where we had multiple hosts with the same unique host id on the same Symmetrix. Fortunately, the DMX is so confused at that point that it won't allow any of the hosts to access the configured devices - and normally your masking and zoning provide some extra protection - but it is still a nasty thing that can happen.

That brings us to the second point. The unique host id can change. EMC will not tell you what changes influence the generation of the unique host id, but for example a change of FC-HBA will cause the unique host id to be changed. On Windows, there are versions of Solutions Enabler where a change in the NetBIOS stack seems to cause this change. Now you might think that you can check what unique host id was configured in the access group, but you would be wrong.

Unfortunately, all the unique host id's that are entered in to an access group will be crypted/hashed by the Symmetrix, and you won't be able to retrieve the unique host id. So my advice. If you want to compare the values you entered, store them somewhere so that you at least have the option to compare the values. It can make troubleshooting a bit easier.

Just as a hint, there is also a way to create static unique host id's, which are unaffected by hardware and software changes. Should you need it, ask your EMC support and refer to Powerlink ID emc198823. They should be able to give you a solution with that ID number. :)

A last word of advice. If you are working with ACL's and changing stuff, please make sure you back up your access logix database before you start with the changes. It might be a good idea to implement that as the first step in any scripts you might create.

ACL's are not a bad thing. They can increase your (sense of) securty. However, the way it was implemented in the Symmetrix environment leaves a bit to be desired, and troubleshooting issues can be a pain if you are not aware of the fact that the unique host id's aren't always unique.

SCSI3 PGR: "Want support on Symmetrix? Reboot 500 Windows servers. Continued.."

By Renegade on Monday 31 August 2009 09:27 - Comments are closed
Categories: SAN, Storage, Symmetrix, Windows, Views: 7.636

Alan Shugart introduced something called the "Shugart Associates System Interface" or in short "SASI" in 1981, and created something that can now be called a commodity. He probably didn't realize back then what an impact his new product would have later on.

You can find the SASI, or SCSI as it is now called, standard in a lot of hardware that is being produced in the storage oriented market today. Among others, you will find the standard in disks used in servers, you will find the protocol in fibre channel SAN's and you will find it being used in high availability cluster environments.

The part about the high availability clusters is the part I want to talk to you about today.

I wrote about HA clustering before and one of the parts that is important when it comes to clustering is consistency in the files used in the cluster.

Lucky for us, the protocol designed by Mr. Shugart (in later versions of the standard) implemented something called SCSI reservations. Basically you can send out SCSI commands like for example the 6 byte reserve command. Earlier versions of the SCSI protocol delivered to us something that a lot of people in clusters call "disk fencing", or SCSI-2.

SCSI-2 is based on exclusive reservations, meaning that only one node owns the disk. This also means that the other nodes can't reserve the disk, which can lead to some "undesired" behavior. For example, SCSI-2 is not reboot persistent. Meaning that a node that rebooted and came up, registered the disk and would be allowed read/write access to it. Not the most elegant solution I would say? :+

Now, SCSI-3 PGR works with group reservations, meaning that every node has a key on a dedicated area of the disk and other nodes can simply remove a nodes key to remove the nodes reservation. It also means that a host will need to register after a reboot, and it will have the option of checking the reservation state. This should avoid multiple hosts having read/write access at the same time, if we don't want them too.

Sounds like a useful feature? It is! :)

Now then, back to our problem with the reboot of 500 Windows hosts. After opening a case with EMC, things went a little dormant. Our host base was verified, and as usual we were asked for emcgrabs/emcreports from every attached Windows host in our environment... 8)7

We checked Enginuity versions on our DMX's and the dreaded support matrices from EMC and found that we really did not have an option, except not upgrading and running the risk of falling out of support.

Right now, the situation if even more tense, since Microsoft came out with a new version of the storport driver in a new hotfix. You can find more info on hotfix 950903 here. The problem being that when you run a HEAT report, this hotfix is recommended by Microsoft. But if the FA flags are not set up in a proper manner, you are bound to run in to problems.

Now, here's a small list of currently required flags for the various operating systems:

Windows Server 2003
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2003 with failover clustering
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2008
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
Windows Server 2008 with failover clustering
  • Common Serial Number (C)
  • Enable Auto Negotiation (EAN)
  • Enable Point-to-point (PP)
  • Host SCSI Compliance 2007 (OS2007)
  • SCSI-3 SPC-2 Compliance (SPC-2)
  • Unique World Wide Name (UWN)
  • SCSI-3 compliance (SC3)
  • PER bit for each clustered device (attribute=SCSI3_persist_reserv)
As stated before, these flags are an absolute requirement to get support from EMC, but unfortunately the situation is still more or less the same. It's amazing how slow things can go along sometimes.

All I can recommend right now is to talk to your EMC representative and explain the situation, and ask for a solution. This will affect people more and more, and in my opinion, this needs to be solved.

Again, I will try to post an update as soon as I have newer information that I can share. Until then I for one am keeping my fingers crossed that we don't run in to problems.

"Want support on Symmetrix? Reboot 500 Windows servers."

By Renegade on Thursday 18 June 2009 10:57 - Comments (13)
Categories: SAN, Storage, Symmetrix, Windows, Views: 6.807

I would dare to say that we have one of the bigger SAN environments here at our company. We have well over 1000 hosts connected to our SAN and use storage from different vendors.

Now, You have your everyday problems when your environment is big. Some problems are smaller, some are bigger, it comes with the territory. But sometimes you just run in to things that will make you think "uhuh, you didn't just say that 8)7 ".

This is the case with a service request that I opened with EMC. I was reading in the EMC Forum when someone made a short mention that the required flags for the Symmetrix front-end ports had changed. I decided to do some checks myself, and found the following document in EMC's Powerlink:
Powerlink ID: emc200609 / "What Symmetrix director flags / bits are required for Microsoft Windows Server 2008?"
Nothing out of the ordinary so far. New settings for a new OS are fine. So just to make sure I also checked for Windows 2003. And I did find something:
Powerlink ID emc201305 / "PowerPath showing loss of connectivity to server down all paths.":
The SPC2, SC3, and OS2007 flags are required flags on all Windows 2003 and 2008 servers connected to Symmetrix arrays.
Now that's something new to me. These were not mandatory before. So I opened a case with EMC and asked them to verify this for me and have them confirm that we need these settings to get some form of support from EMC. I received a longer mail back, but the third sentence in the mail stated the following:
Your observation is correct.
If you check the current ESM document, you will find that these settings are mandatory. To top it off, the mail stated that:
Please note, that setting the flags changes the inquiry page and hence the PnP id. You will need to reboot.
This means that we can make the change on the FE-port, or we could set the flags on an initiator base, but each way we would need to reboot about 500 hosts connected to the various Symmetrixes.

We are still talking to EMC to find another solution, but this issue is not an easy one. In all fairness it should be said that this decision is not EMC's fault. Microsoft changed the requirements for Windows 2008 hosts, and just said they want vendors to use the same flags for Windows 2008 and Windows 2003. The result being the issue described above.

I'll update this post or write a new post as soon as we hear anything more, but this is turning out to be an interesting change. Let's see what happens.

EMC World 2009: Day 2 recap. It's raining again...

By Renegade on Thursday 21 May 2009 15:03 - Comments are closed
Categories: EMC World 2009, Symmetrix, V-MAX, Views: 1.900

Seems to be some record breaking weather that we are seeing here, and Orlando seems to have some problems coping with the amounts of water that are falling out of the skies.

So, what happened here? Well, there were two main topics that are both related to the new systems that was introduced a while back by EMC.

Enginuity 5874

There was a session on the changes in the new Symmetrix Enginuity version 5874 for the new V-Max system. We were informed that among others the RAID-engine was redesigned. The new version allows for only 1 mirror per device instead of the previous 4 mirrors per device. They now allow all of the features on all of the RAID-levels, and they introduced the possibility of VLUN migration.

Also new was the support for large volumes. The limit increased from 16GB to 240GB. This should help a lot of people who were required to create large metasets before. Also, you can now create 512 hypers per drive.

You will find some changes in the management and provisioning section for this new version of Enginuity. Dynamic provisioning received some enhancements that reduce the number of steps required to present your storage to your host. You now create a storage group, a port group and a device group or initiator group. After that you are basically done. Create some templates once these prerequisites are created and you can present your storage with relative ease. People will also find that bin file changes are speeded up on this new release.

SRDF also received some upgraded numbers. The maximum number of SRDF groups is now at 256 groups per array, and 64 groups per FA-port. Still, you might want to reconsider splitting up all groups on 4 FA-ports. ;)
They also introduced the new so called "DLDEV" or Diskless R2 Device, or in Symmetrix speech a so called R21 device that allows you to cascade to a 3rd site using SRDF/A. DLDEVs need to be on a V-Max though.


After an interruption I got in to a private V-Max session a bit later then I planned, but I still managed to get some new information.

Currently one of the less official recommendations that you will find is that if you have a large datacenter, you might consider distributing some engine groups in the various areas of your datacenter. Current distance limits you will find are at around 100 meters. Simple reason for this distance limitation is the fact that you are working with latencies. If you take the distance up to 100 meters, you will find that latency will increase with a factor 27. Upping the distance could create some unforeseen issues.

One of the questions was that with FAST we will see a tiering over the various storage media, but what will EMC do to tier my storage on the various storage classes. In a normal situation you want your mission critical data on high end arrays, and work your way down. One could imagine something similar to FAST on a different level that would allow you to better utilize the various availability levels of you storage hardware.
The engineers implied that they are working on this, but unfortunately would not go in to a great deal of detail.

One of the most heard questions so far is probably "When will FAST v2 be coming to the V-Max?". Current timeline indicated about 1 to 1.5 years, but no one was willing to talk about any of the technical details. This raises the question how far they actually are with the FAST technology, and that's a question that produces the same "no coment" remark and look. What they were able to tell though was that they are planning to introduce FAST on the Clariion series array, which will be a major money maker when they are able to deliver. Currently you will find around 300,000 CX's worldwide, and one can only imagine how many people might be interested in purchasing this technology, if and/or when it works. A first version of FAST v1 should be introduced at the end of the year according to Barry Burke.

And even on a V-Max the newer FAST version will bring in some major cash. You will find it as a separately billed option when you would like to use FAST for fat LUNs and thin LUNs.

However, one general statement was clear. "FAST is coming!"

Howdy Y'all, I'm a newbie

By Renegade on Friday 17 April 2009 10:28 - Comments (6)
Categories: EMC, HDS, SAN, Storage, Symmetrix, USP-V, V-MAX, Views: 3.949

To blogging that is. :)

To start my first post I need to get something off my chest. Anyone who has been involved with SAN has probably noticed the launch of the new Virtual Matrix Architecture (or in short V-Max) by EMC last Tuesday.

I am still busy trying to gather all of the information and filling in the blanks that still exist for me. Anyway, all in all to me the solution so far looks pretty good and will need to prove itself in terms of scalability and such.

What really bugs me though is the reaction of some competitors (actually, at this moment only one competitor). The culprit? Hitachi Data Systems or in short HDS.

And why you might ask? Well, to me it seems that they felt the need to respond quickly. At least they did so, I'll give them that much. But instead of replying they tried to start a good old flame war. All of the classic signs are there. No knowledge of the matter was pretty poor and they tried to score by fussing over the name of the product.

All in all, as a customer and end-user of the various SAN solutions I have to say that the replies made by the various HDS people made me feel less likely of possibly purchasing a USP-V from them.

Me, I like to see products that solve the problems that I have in a reliable way. Be it from an infertile tiger or from an electronics repair shop. I just need to get the job done, what do care about the name? ;)