Unfortunately in my line of work as a consultant, I am contacted far too often by resume hoarders, who are “recruiters” who sole job is to collect resumes, with no attempt at all to try to match the right person with the position that is open.  It’s just a matter of collecting resumes and the persons required pay rate and passing them all onto the client.  No vetting to see if you have the required skills, no seeing if you have the people skills – you are just a number and they just want to spit out a million resumes to the client.  It’s all about volume (which will become evident when you see their low markup).  Even worse is when the client is not the end-client, but rather a middle-man that adds another layer in the process (i.e. who is called an “implementation partner” but is just another recruiting firm).  In essence, you have a company that is just a resume hoarder working for a recruiting company who does not want to bother with finding people on their own.

It is a tiered system.  Prime contractors have the direct relationship with the customer.  Subprime do sourcing work for the prime contractors, taking a small markup.  Sub-subprime (and so on) do work for the subprimes, like collecting resumes for people that haven’t posted theirs on dice, monster, etc. These are the resume hoarders I am referring to.

These resume hoarders are notorious for sending out spam: wading through thousands of resumes and LinkedIn profiles for a keyword and mass mailing those unfortunate people with that keyword.  Their hope is out of thousands of emails they will get 1-2 people who don’t know any better and will submit their resume.

Below is a recent conversation that I had with one of these resume hoarders, who was working for a company called TM (not their real name of course).  I have had contacts with many other companies who were “implementing partners” of TM.  TM likely looks at the resumes with the lowest rate and submits them to their client.  It’s all a numbers game with no desire to determine what is best for the client and what is best for the consultant.

With these firms you won’t get any insight on where you stand in the process and if you get declined there won’t be any feedback.  There is no client connections/value/help in understanding the job opportunity, and this will just turn in to a wild goose chase and a waste of your time.  Your resume will just go in to a black hole and there will be minimal accountability.

Here is the email chain (rates are fake):

Dear James
Hope you are doing well,
Please find job description below and let me know if you are available immediate for below position.
BI Lead
Houston, TX
1+ Year
(details omitted)

>> What is the pay rate?

Rate is open please let me know what rate are you expecting?

>> Find out for the client what the pay rate is and I’ll let you know if I’m interested.

Ok we do have a number $50-60/hr on corp to corp all-inclusive. Please let me know if you are ok with this to move further?

>> Sorry, my rate is double that. You will need to look for a junior-level person.

Ok please let me know your number I can pay you that rate as I told you my number?

>> As I said, my rate is double that, which is $100 – $110, way out of your range.

No we can pay you $110/hr on corp to corp all-inclusive and its with Client X, Please send me your updated resume with contact details ASAP. [My note]: They will say you can be paid whatever you want to convince you to send them your resume

>> You tell me that the client is willing to pay $50-$60/hr, and now you say they are willing to pay double that? You will need to explain that, as I don’t send me resume and contact details unless I know the client is aware of my rate and there is a good chance at an interview. Too many resume collectors out there that waste my time. Is Client X your direct client?

I told you already that rate is open and you told me to find out and I have given you number $60/hr. So the rate is open we can pay you $110/hr and we are going through with our Implementing partner we are not working with direct client. Please let me know?

>> Who is your implementing partner? Can someone from that company contact me about the position?

It’s with TM. And no one can contact you for this as they just get the requirements from Client they not do recruiting like us.

>> Thanks for the clarification. So if I give you my resume, what happens next?

We will submit to our implementing partner and they will contact you through us and they will try to schedule your interview with Client X ASAP. Please send me your resume to move further.

>> Great, thanks. One more question before I submit my resume: What is your markup rate, and what is the rate you will bill the client?

Only $3/hr will be margin on your submittal and if client will hire’s you. Please send me your resume I appreciate. [My note]: So they get $3/hr, then TM adds their markup, and that determines the client bill rate.

>> Aren’t you or TM going to interview me first to see if I am what Client X is looking for?

TM will be do this. Can you please send me your resume our work is just to submit your resume to them they will take care of your hiring process.

[My note]: If you do submit your resume, they will call or email you to request info about yourself (usually the last 4 digits of your SSN).  Then they will want you to reply to an email to confirm your pay rate.  This email will indicate you will be contacted by another recruiter to be given more details on the opportunity.  Then you will never hear from them again.  I have done this process dozens of times and never once received any further communication at all, in large part because of my rate as they try to find junior-level people or visa-sponsored people who will accept a low rate.  There is no thought given to what is best for the client.

I asked my recruiter friend about this.  His response: “Run.  Ran away as fast as you can!”

This post is basically to answer a question asked in class this week: How can we get the last statistics update date for ALL user tables in a database? After working on the query for a while, I realized that the new metadata function I posted about here...(read more)

Hey guys,

my session “Troubleshooting SQL Server with PowerShell” is available to download :

http://sqlbits.com/Sessions/Event11/Troubleshooting_SQL_Server_with_PowerShell1

Hope you guys liked !!!

Have you heard about Hadoop but don't know much about it? What about "big data?" Would you like an intro at the 20,000 foot level that won't take more than an hour of your time?

 

Jeremiah Peschka (blog | twitter) was recently on .NET Rocks, a podcast that publishes twice a week which focuses mainly on technologies of interest to Microsoft-centric developers. You can find his talk here:

 

Show 898 - Big Data with Hadoop with Jeremiah Peschka

 

If you're not familiar with .NET Rocks, they've recently crossed over 900 shows and are on their way to show #1,000. I've listened to the podcast on and off for years and have always gotten a lot out of it. If you haven't listened to it, it's worth giving a shot.

The first thing that I want to say about Tim’s response is that it would have been better – much better – to have included this in the email that was sent out announcing the decision to terminate the advanced certification programs. You will find several others have responded with the same sentiments. While this does not excuse this type of notice being sent out to all of the current advanced certification holders by email, or in the dead of night, at the start of a 3-day holiday weekend (that action just reeks of trying to avoid a backlash from the community), it does a much better job of explaining the “why”. Even if some of the “why’s” being given are suspect…

So, on to my responses:

Lack of traction

Tim states that these programs haven’t gained the traction hoped for, and cites a 0.08% of all MCSE certified persons has having the advanced certification. So, why is this?

The MCSE process is so easy, that people that shouldn’t have this qualification have it. This is a brain-dump certification. So many people have acquired this certification in this manner that it is a worthless paper certification. Its value to all has been diluted by so many obtaining it by cheating the system. As I mentioned in my previous blog post, and in my opinion, having the MCSE means only that the person with that certification has been exposed to the technology covered in the MCSE. It does not convey competence, or even experience, with it. So, Tim, start here. Make the MCSE tougher. By having a higher quality prerequisite to the advanced certifications, you will have fewer persons with it. So the percentage of MCSE to MCM will be much higher.

Because the MCSE is so easy, people that want to achieve the advanced certifications have a lot of work to do to be prepared for being a master. Most professionals at work only have to contend with the issues present at their employment – a MCM has to have a broad range of expertise across the product. This will require a lot more effort to learn and prepare for the MCM than what you will get on the job. Considering that most will be doing this preparation and learning on their own time, it will not be a short course. Many people that responded to the Connect item, or have blogged about their preparations for the MCM, are talking about preparation in terms of years. Yes, that is plural. And it’s no joke – this is not a short-term commitment. Excellence takes a while. If you pause and think about it for a minute, this is what we want for the advanced certifications – people that have been there, for quite a while, with an in-depth knowledge of the systems.

Just how many of the MCSEs, with that brain-dump that they did to obtain that certification, have the drive or passion to further themselves? How many want to go to the master level? Most will stop right there at the MCSE level. Why? Because they don’t have the drive or ambition to devote the required amount of time to learn the required skills to be at the master level. Because working with SQL Server (or any other platform) is a j-o-b. Not their passion. Becoming a MCM is too hard, and they can’t cheat their way into it – otherwise they would. So they don’t even try.

Another thing to consider for the lack of traction – Tim also states that many of the advanced certifications are out of date, and cites the specific example of SQL Server 2008. Yes, the current master level for SQL Server is only for SQL Server 2008. With a previously announced deadline (http://blogs.technet.com/b/themasterblog/archive/2012/09/11/mcm-microsoft-sql-server-2008-knowledge-exam-retirement-date-announced.aspx) of January 1^st, 2014 for the SQL Server 2008 knowledge exam, how many people are waiting for the new SQL Server 2012 exams to become available before entering the program? I personally know of several, and I can’t blame them.

Considering the large number of paper MCSEs, and the low percentage of MCMs, I’d say that the MCM program is working as it should be – it is only for those that can prove their stuff, and getting rid of the fluff. The MCM program is fine – it’s the rest of the certifications where the problem is really at.

However, I have a suggestion for how to change things here. Be patient, I’ll reveal it shortly.

Lack of knowledge of the existence of the advanced certifications

The advanced certifications are not emphasized anywhere. When I present, I mention that I am a SQL Server MCM – and frequently I have to pause and explain just what that is. The industry doesn’t know. Not the SQL people hired to do the work. Not the managers. Not the businesses. Not the HR staff. Not the job recruiting companies. Almost nobody has heard about this program, or knows what this program is and what it means. And the onus for this comes back to Microsoft. If the industries don’t know about it, it is solely because there hasn’t been enough from Microsoft to educate them. There is no emphasis of the advanced certifications in other programs.  There is no marketing of the advanced certifications or their  programs. In short, there is nothing coming from Microsoft saying that one ought to pursue advanced qualifications.

Greg Low remarked in his posting in the Connect item two things that could be done to improve upon this, and I completely agree with both of these. Allow me to quote those here:

A first step in making the program more relevant would have been to make use of it within other Microsoft programs. For example, Microsoft Certified Trainers could have been required to have at least taken the knowledge exam. When I suggested that, it was met with howls of “but we’d lose most of the trainers”. While that’s partly true, what does that tell you about the existing trainers?

I think that this is an excellent suggestion. I’m surprised that one can be a MCT without a higher certification level than what is currently required of them. We need trainers that know their stuff. If they can’t pass the knowledge exam, they shouldn’t be training others. I sure wouldn’t want them training me, or anyone else that I know.

Greg also suggests:

Instead of abandoning it, why not take quality seriously and see it applied throughout related programs. The MCT program is one but another would be the Gold Partner program. Is it really acceptable to have Gold Partners working (and promoted by Microsoft) on the largest projects, without a single MCM/MCSM on staff? Why not require real demonstrated ability in the Gold Partner program?

Again, this is something that I agree with, and not only for the reasons that Greg lists. In addition, it will drive up exposure of the advanced certification programs. As the Microsoft Gold Partners start getting their employees certified as MCM, they will start promoting this to the businesses that they deal with. They will get business from companies that want, even demand, that MCMs be on staff to support them. This will in turn drive up exposure throughout the business. Managers will know about this program. Workers will become knowledgeable about this program. When these businesses start looking for applicants, they will start requiring a MCM. Recruiters will become aware of what a MCM is (I have yet to meet a single recruiter that knows about the MCM program for any platform!).

My Suggestions

With this background, I present my suggestions to improve things with the advanced certifications:

1. As I have outlined above, my opinion of the MCSE certification is that it is worthless, and only suitable as an indicator of exposure to a product. Therefore, the comparing of the MCSE to MCM is also invalid. However, revamping the MCSE would become confusing (is that certification earned before or after the new requirements?).
   
   Instead, what I propose is to create a new certification, to be positioned between the MCSM and the MCM. Keep the MCSE as it is. I suggest calling this new certification level the “Microsoft Certified Journeyman” (MCJ). If we look at the definition of Journeyman, I think that it applies here perfectly. When we further look into the history of a Journeyman (from Wikipedia), we have “A journeyman is someone who has completed an apprenticeship and is fully educated in a trade or craft, but not yet a master.” In the USA, in the building trades, it usually has a time-in-field requirement. I think that a minimum amount of time as a MCSE should be included in the MCJ. Perhaps this time-in-field requirement should be coupled with the existing MCM Knowledge exam to be requirements for the MCJ.
   
   The comparison should then be between the MCJ and the MCM.
   
   A few people have talked about the MCJ before – see the “Stepping Stone Cert” articles at http://jasonbrimhall.info/tag/stepping-stone/ for more details and other links. (Note that some of the specifics mentioned in these articles were with the intent of this certification not being sponsored by Microsoft.)
   
   To keep a synergy going with the certifications, perhaps rename MCSE or MCITP to be “Microsoft Certified Apprentice”. (The problem with this is that there already is a MCA – and it’s an advanced certification.) This will help focus things on the Apprentice to Journeyman to Master progression. Just don’t rename the MCSE to the MCJ – as the certifications stand now, the MCJ needs to be between the MCSE and MCM!
2. Initiate a new policy for the MCT certification – as of July 1^st, 2014, they must be a MCJ to retain/earn their MCT certifications.  That gives them 9 months. If they are a good trainer, this should be just a scheduling issue for them to take the exam. If they don’t have the knowledge, well, then in my opinion they shouldn’t be a trainer.
3. Initiate a new policy for Microsoft Gold Partners – as of January 1^st, 2015, they must have at least one MCM on staff to be a Microsoft Gold Partner. This gives the companies 15 months to get someone certified before they lose their Gold Partner standing.
4. Microsoft needs to keep the MCM program current.
5. Microsoft needs to promote the MCM and MCA in all ways possible, and specifically in how they can really ramp up service to a customer when the customer utilizes a person with advanced certifications. Heck… I frequently see job postings looking for a MCDBA (for those that don’t know, this is a certification for SQL Server 2000). There is a serious problem here with all involved in knowing what the certification process actually is.

Other notes:

Tim’s posting on the Connect article still leaves several unanswered questions.  Foremost amongst these is the apparent discrepancy in the timing. He states that this decision was made over the course of many months of deliberation. Yet during just the past month, there have been events that convey the opposite:

1. Microsoft has accepted applications for inclusion into the advanced certification rotation schedule (see the remark from Heath Groves in the Connect item).
2. The expansion of Prometric testing centers after de-coupling of training and exams (http://blogs.technet.com/b/themasterblog/archive/2013/08/12/you-asked-and-we-listened-increased-number-of-prometric-test-centers-for-mcm-mcsm-exams.aspx).

Tim also talks about the high cost for the participants ($20,000 USD). With the de-coupling of the testing and training, this cost is now much lower. The SQL Server platform has been decoupled for a while now, and since it has been decoupled, there has been an upsurge in people obtaining the MCM – myself included. The above link implies that the de-coupling occurred for the other platforms very recently. Since the training has been decoupled, this is no longer available only in the USA (the exams can be remotely proctored, and the SQL Server platform has had this as an option for quite some time now). The only part of it that is “English-only” is the exams – surely these can’t be that difficult to translate?

After debunking virtually everything that Tim has stated, what is left? Either, as several have opined in the Connect article, Microsoft wants everything as a *aaS service, and the advanced certifications were enabling on-premise activities, or cost. While I wouldn’t be surprised in the least if it’s the former, I strongly suspect it is the latter. Tim did make several references to the expense of the program (emphasis mine):

Sure, it loses us money (and not a small amount)

And many of the certifications currently offered are outdated – for example, SQL Server 2008 – yet we just can’t afford to fully update them.

To me, this just reeks of a cost-cutting measure. I think that Microsoft Learning has a line item in their budget called “Advanced Certifications”, and it’s large. Perhaps they should revamp their budget by product, and each product includes the advanced certifications. I hope that I’m wrong, but I don’t think so.

In this case, the expression “penny-wise, pound-foolish” applies. I would venture that folks with advanced certifications are driving many operations that net Microsoft a lot of money in licensing fees. I predict that dropping the advanced certification program will “lose” Microsoft more money than what the cost of administering the advanced certifications program is.

Is it time to stop being a DBA?

Unfortunately, this decision has already caused some people known to me to abruptly decide upon a change in their career. My fear is that there are more to follow. To those that are considering doing so, I request: Please give Microsoft some time to come to their senses and straighten this situation out. Please don’t make any sudden, rash decisions. Please wait until at least January 1^st, 2014 before making a major decision such as this.

Media Coverage

I’m glad that the media has been picking up this story. So far, I have seen articles from The Register , ZDNet, Windows IT Pro, The Channel, IT World and another from The Register. I’m hoping that more media outlets will pick this up, and as they do so and have articles about this, I will update this post to include them. My hope is that the major networks in television and print all pick this up and run with it.

Community Response

There are a lot of people in the various products that are posting their reactions. Instead of trying to find and post each person here, this Google search has been quite useful to me in finding others thoughts: https://www.google.com/search?q=MCM+advanced+certification+retire

(I’m sorry if you prefer the Microsoft search engine, but right now I’m just not going to support it, or even mention it by name.)

MCM4Life

Microsoft isn’t revoking the advanced certifications, so those that have earned it carry their title for life. And I would still have my license plate. (A special thanks to Robert Davis for the MCM4LIFE graphics!)

Keeping in touch

If you are going to comment on this action by Microsoft, please use the twitter hashtags #BringBackMCM and/or #SQLMCM.

[Today's post comes to us courtesy David Fabritius from Windows Server Marketing]

In this article, I’ll explain a licensing change that has been made to Windows Server 2012 R2 Essentials as well as some related licensing topics when using the new Windows Server Essentials Experience role with the Standard and Datacenter editions of Windows Server 2012 R2.

The Hyper-V role and Hyper-V Manager console are now included with Windows Server 2012 R2 Essentials! The server licensing rights have been expanded, allowing you to install an instance of Essentials on your physical server to run the Hyper-V role (with none of the other roles and features of the Essentials Experience installed), and a second instance of Essentials as a virtual machine (VM) on that same server with all the Essentials Experience roles and features. Previous versions of Windows Server Essentials (and Windows Small Business Server before that), made it necessary to obtain a hypervisor separately, such as the Microsoft Hyper-V Server, but that is no longer required. In addition, we are working with our OEM partners to help them offer server hardware pre-installed and pre-configured with Windows Server 2012 R2 Essentials running as a virtual machine.

A related licensing question that is often asked when virtual instances of Windows Server are used is around license assignment, especially in the case where you are using either the Live Migration or Hyper‑V Replica feature to move VMs from one Hyper-V host to another. The most fundamental thing to keep in mind is that Windows Server licenses are assigned to physical servers, not to virtual machines. So when a VM is moved, the Windows Server license assigned to the source server does not move and is not reassigned. With that in mind, the destination server must be licensed appropriately to run the VM that you moved to it—as well as any VMs it was previously running. For example, if you purchase a server from your favorite OEM that is preinstalled with Windows Server 2012 R2 Essentials running as a VM, there is no problem with migrating that VM to another server that is running Windows Server 2012 R2 Standard as long as the destination server is not already running all of its allowed virtual instances. For more information about how to license virtualized environments, see this Licensing Brief.

Next I’d like to discuss a few licensing implications for the new Windows Server Essentials Experience role, which is now an option available in the Standard and Datacenter editions of Windows Server 2012 R2 (it is turned on by default in Windows Server 2012 R2 Essentials). The first implication is around downgrade rights from the Standard edition to the Essentials edition, which was introduced with Windows Server 2012. While this option remains available, if you are only interested in gaining access to the value-add feature set of Essentials, you no longer need to downgrade but can simply turn on the Essentials Experience role.

Finally, it is important to note that turning on the Essentials Experience role does not change the licensing terms for the edition that you purchased in any way. The most common question is around using the Remote Web Access feature of the Essentials Experience role. Because this feature makes use of the Remote Desktop Gateway role service of Remote Desktop Services (RDS), an RDS client access license (CAL) is required for each user who is using this feature with the Standard and Datacenter editions. However, for the Essentials edition, as with previous versions, the use of Remote Web Access does not require an RDS CAL. For more information, see this Licensing Brief.

If you have any additional questions or comments about Windows Server 2012 R2 Essentials or the Windows Server Essentials Experience role, please join us in our support forum.

How to build a Failover Cluster on a couple of laptops

This is a continuation of our playing with (learning) Failover Clustering for our evaluation of Windows Server and System Center.

Note: This is NOT best practice deployment advice. This is “notes from the field”. This is how to get by with what we have.

If you have been following along, you have a couple of domain joined Windows Server machines running Hyper-V and have a Failover Cluster running on one of them.  Our Failover Cluster is pretty basic as it currently stands – two virtual machines accessing shared VHDX files. 

Before we continue, did you get to “play” with your Failover Cluster? Did you try adding another Network Adapter (and configure a Heatbeat network)? Adding disks to Clustered Shared Volumes? How about adding more disks and messing with Storage Pools? Two and Three-way mirrored Virtual Disks? Creating a File Server?  A Scale-Out File Server? An SMB Share (for Applications)?  SQL Server anyone? Anyone brave enough to add another node??

Let’s build a Failover Hyper-V Cluster on laptops

OK, lots of ways to skin this – and all depending on what hardware you have access to.  I’ll do it with two laptops, a gigabit switch, four USB to Ethernet dongles and a bunch of cables.  If you have three laptops, just put the Domain Controller and the Storage onto the third one.  If you don’t have any USB to Ethernet dongles, you can do this with just the one network.  Three laptops is the best solution, as it will let you “kill” a cluster node and watch everything fail over automatically (you can do this with two laptops – you just need to be careful as to which one to switch off).

Compute

We already have two domain joined laptops, running Windows Server with the Hyper-V role enabled, so let’s start from there.  We also have a Domain Controller running.  So, that’s the compute part of our environment sorted.

Network

We are already using the laptop’s Ethernet port for both the Management network and as the Hyper-V virtual switch for our VMs to communicate over. We need another network for the Cluster Heartbeat and another one for Live Migration. In the “real world” we’d also have either Fibre Channel, or SAS to the storage OR more network interfaces for iSCSI. And we would be using NIC Teaming and/or Converged networks for high availability (I’ll post something on both of these at a future date as they are both “laptop-able”).

I’m going to use two USB to Ethernet dongles in each laptop (Heartbeat and Live Migration).

It is a best practice to name your network connections – it makes your life so much easier.

I’ve configured my Heartbeat network to have 10.0.0.0/24 addresses and Live Migration has 20.0.0.0/24

Storage

This is definitely NOT a best practice and is really only for our evaluation purposes.  Mind you, if you have a third laptop (or old PC or Server), then you could continue to run like this (with separate storage).  If you only have two laptops, then your clustered storage will have to be on one of them – and that means the path to your storage has to go out over the network, through a virtual machine and through Hyper-V before it hits the disks (which means that it is not optimal).

There’s an easy way to do this and a hard way – we’ll do both!

Easy is a File Share, hard is an iSCSI Target.

You would think that in both cases you’d just enable that storage on one of the laptops – you would be wrong!

• Failover Clustering is clever enough to realise that the SMB File Share is on one of the nodes (and that it is therefore NOT shared storage) – try it if you like, just to see the error message.
• The iSCSI initiator can be configured to loopback onto itself, but it really doesn’t work very well at all (so I’m not going to tell you how to do it)!

You’re going to need another Virtual Machine – I’ve called mine \\STORAGE and have added the File Server and iSCSI Target Server roles.

Ideally put \\STORAGE on some fast disk – SSD or external caddy.

The easy way (an SMB File Share)

• On \\STORAGE create a share (c:\VMs).  Give full control to both the laptop machine accounts.
• In Failover Cluster Manager, create a cluster.
  • Add the nodes
  • Give it a name (HV-Cluster)
  • Give it an IP address if not using DHCP
  • Make sure you un-check the Add all eligible storage to the cluster box
  • Click Next

• Your cluster is built – but has a warning about the Witness
• Configure the cluster (in Failover Cluster Manager)
  • Go into Networks and rename them – to match the names we gave them earlier
  • Specify whether they are to be used for cluster communications and/or client
    • Click on the Network Connections tab at the bottom of the page – it all makes sense!

• Right click on Networks and configure Live Migration Settings

• Configure Cluster Quorum Settings
  • In Failover Cluster Manager, Right click your cluster (HV-Cluster.contoso.ie), select More and Configure Cluster Quorum Settings
  • Select Advanced quorum configuration
  • And Configure a file share witness and point it to the Share you created at the beginning of this section (\\STORAGE\VMs)
• Your Hyper-V Failover Cluster is ready (patients – we’ll play soon…)

The hard way (an iSCSI Target)

There is some preparation work to do before we can create our cluster:

• In Server Manager on \\STORAGE go into File and Storage Services –> iSCSI and start the New iSCSI Virtual Disk Wizard
  • Enter a custom path (create a new folder to keep things tidy – c:\iSCSI)
  • Create a 1GB Quorum disk (go for Dynamically expanding – we’re using laptops)

The wizard will then guide you through creating a New iSCSI target

• Give it a name (Storage)
• Add both the laptops as iSCSI initiators
  • Do it by IQN (as it will work 100% of the time – I’ve had “issues” with both IP Address and DNS Name)
• For our evaluation, leave the Authentication alone (we don’t need it)
• Click Create (ignore the error)
• Create a second iSCSI Virtual Disk (30GB, Dynamically expanding)
  • c:\iSCSI\VMs
  • Assign it to the existing iSCSI target (that we created earlier)

We can now get our Hyper-V hosts to connect to the iSCSI Target

• On BOTH laptops, start the iSCSI Initiator
  • Say Yes to have the service start automatically
  • Type the name of the iSCSI server into the Target box (storage) and click Quick Connect
  • Assuming all is well (and it connects) click on the Volumes and Devices tab and click Auto Configure

You should see two entries in the Volume List (these are the two virtual disks that the Target is presenting)

• On one of the laptops, go into Disk Management and online and initialize the two disks then create NTFS volumes (don’t assign drive letters to them).

C:\ & D:\ are the two internal SSD drives in my laptop.  Disk 2 & Disk 3 are my iSCSI attached drives.

You are now ready to create your Failover Cluster

• The process is exactly the same as “The easy way” above – except DO check the Add all eligible storage to the cluster box

• Configure your Failover Cluster as “The easy way” above, but in addition:
  • In Disks, you should see that the smallest shared disk has already been configured as the Disk Witness in Quorum (the 1GB drive)
  • Right click on the other drive (Available Storage) and Add to Cluster Shared Volumes (this makes it available for Application and Hyper-V servers to use)
  • Optionally rename the disks (as it just makes your life easier)

Let’s create highly available Virtual Machines

If you already have a bunch of VMs running on your hosts, you can make one of them highly available.  You won’t be able to do it to your Guest Cluster nodes (not yet anyway – try, just to see the error) and I don’t suggest you do it to your Domain Controller as it will make things awkward later on.

• In Hyper-V Manager, right click the VM you want to make highly available and select Move
• Select to Move the virtual machine’s storage and then to Move all of the virtual machine’s data to a single location

Your VM can be running – this is a Live Storage Migration

• If your cluster is built with iSCSI, then the location for the virtual machine’s items is C:\ClusterStorage\Volume1\(optional sub folder)
• If your cluster is built with an SMB File Share, then the location is \\STORAGE\VMs\(optional sub folder)

Now you can run the Failover Clustering High Availability Wizard

In Failover Cluster Manager, right click Roles and select Configure Role Select Virtual Machine And then select the Virtual Machine who’s storage is now on the shared storage (iSCSI or File Share) Confirm and Finish Your Virtual Machine is now running as a Failover Cluster resource and all management should now be done from within Failover Cluster Manager.

To build a highly available VM from scratch, select New Virtual Machine from within Failover Cluster Manager.  Select the host where it will start its life and run through the New Virtual Machine Wizard.  Make sure you select the shared storage for both the Location of the virtual machine and its virtual hard disk.

You now have a two –node Hyper-V Failover cluster running on two laptops.  And have at least one highly available virtual machine.

Let the learning begin!  Right click the Virtual Machine and see what you can do to it (e.g. Move, Live Migration vs Quick Migration).

Enjoy,

@DaveNorthey

Part 1 in this series is here: Introduction

Part 2 in this series is here: Fabric

Part 3 in this series is here: Failover Clustering (part 1)

Part 4 in this series is here: Failover Clustering (part 2)

I didn’t want to write on this sudden happening, but then I couldn’t hold my anguish after reading so many posts, articles, and I thought I should share my quota of wailing. Two things broke my decision – an InsiderMail from Paul Randal (@PaulRandal | Blog) on 9/3, and the post from Satya (@SQLMaster | Blog) on 9/3. Both the industry experts came up with their feelings on the sudden happening, coincidentally on the same day.

MCM is discontinued and this is what gave me a shocker, a stunner. This post talks about on how Microsoft has decided to get it off its list. I kept spending few hours reading different articles, posts, news online, news feeds with the key words MCM and realized I am hearing, and actually bearing the brunt of the fact that Microsoft has decided to retire the MCM program. If I have to be honest to myself, I was trying hard to get closer to my dream and goal of obtaining a MCM. Now looks like I wasn’t getting anywhere near, apparently, when something has holding me back – retirement of MCM on 1-Oct-2013. Not a good news at all for folks out there who have probably dreamt like me, and put in more hours to get one or trying to get that ultimate achievement. Microsoft has denied that feeling to all those fans of SQL Server who worked years on the product, by just snubbing with a pull-off-the-plug action, on the program. The death of MCM is going to be quicker than retiring the famous reader – Google Reader. It took Google few months to bring down the service, and Microsoft is going for the death-punch in less than 2 months, huh. A known play as usual, and one without any breather by Microsoft. Nice job, uh.

I could go on ranting my displeasure and disappointment on the news, but then what’s the use of crying for what has happened. The work would get back to usual after few days everywhere, and everyone will visit their priorities. However someone has to keep up the fire and choke the guys who have done this abrupt thing. So go ahead and please, please vote up for a connect item on MSFT website created by Jen Stirrup – a SQL Server MVP protesting and asking back the MCM program.

One last thing – another blogger has shared his displeasure on this post, and made sense to me http://michaelvh.wordpress.com/2013/08/31/microsoft-is-retiring-the-mcsmmca-program/

If you happen to read this post, please share with as many as possible so that there is movement to reverse the decision or probably come up with something better than MCM at least. Don’t forget to vote-up if you don’t want to share this post.

Thanks for reading the bereavement news.

Sad Feeling – Shyam Viking.

I’m sure most of you have heard the news now that Microsoft has effectively cancelled the MCM/MCSM/MCA program abruptly, with a cold email sent at 10pm last Friday night. There’s a lot of anger in the community about the way it was done – with only one month’s notice for people to attempt the exams before everything closes down October 1st.

The Connect item to bring the certifications back had almost more than 850 up-votes before it was closed and made inaccessible outside of Microsoft (due to abuse from a small handful of people) and there are lots of impassioned comments berating Microsoft for cancelling the program. Plus many people have blogged their thoughts – I liked Jason Brimhall’s post that I linked to in my commentary in yesterday’s SQLskills Insider newsletter.

[Edit: the survey is closed - results are here.]

I think it would be interesting to know just how many are affected by this decision, as I know that many of you have invested time studying for the MCM and money and time attending classes like ours.

And that’s the point of this survey. I’ll editorialize the results next week. You’ll notice there’s only one “No I’m not affected” answer – that’s because I’m only interested in the number of unaffected people.

Thanks!

The post Survey: SQL Server MCM cancellation – does it affect you? appeared first on Paul S. Randal.

In an effort to migrate as much content as possible from SQLCAT.com to the MSDN SQL CAT site we have consolidated all of the Whitepapers published over the last few years.   While all of the content actually exists on Technet and MSDN download sites, SQLCAT was a great resource to start looking.  Hopefully this will make finding those papers a little easier.

The organization is not quite perfect as many of these papers can cover multiple features and topics, but this is pretty close.  If you happen to find any links that are broken or wrong, please post a comment and we will get it addressed. 

More content will be migrated by the end of month, so stay tuned!

 

SQL Server High Availability Backup Recovery

SQL Server 2012 AlwaysOn: Multisite Failover Cluster Instance

Microsoft SQL Server AlwaysOn Solutions Guide for High Availability and Disaster Recovery

Migration Guide: Migrating to SQL Server 2012 Failover Clustering and Availability Groups from Prior Clustering and Mirroring Deployments

AlwaysOn Architecture Guide: Building a High Availability and Disaster Recovery Solution by Using AlwaysOn Availability Groups

AlwaysOn Architecture Guide: Building a High Availability and Disaster Recovery Solution by Using Failover Cluster Instances and Availability Groups

A Technical Case Study: Fast and Reliable Backup and Restore of Multi-Terabytes Database over the Network

Database Mirroring and Log Shipping Working Together

Database Mirroring Best Practices and Performance Considerations

High Availability and Disaster Recovery at ServiceU: A SQL Server 2008 Technical Case Study

High Availability and Disaster Recovery for Microsoft’s SAP Data Tier: A SQL Server 2008 Technical Case Study

Failure Is Not an Option: Zero Data Loss and High Availability

Implementing Application Failover with Database Mirroring

Proven SQL Server Architectures for High Availability and Disaster Recovery

SQL Server 2008 Failover Clustering

SQL Server High Availability and Disaster Recovery for SAP Deployment at QR: A Technical Case Study"

SQL Server Replication: Providing High Availability using Database Mirroring

Partial Database Availability

Using Replication for High Availability and Disaster Recovery

 

 SQL Server Performance

Analyzing I/O Characteristics and Sizing Storage Systems for SQL Server Database Applications

Predeployment I/O Best Practices

SQL Server 2005 Waits and Queues

Database Snapshot Performance Considerations under I/O-Intensive Workloads

High Performance SQL Server Workloads on Hyper-V

Performance Guidance for SQL Server in Windows Azure Virtual Machines

Loading Bulk Data into a Partitioned Table

Running SQL Server 2008 in a Hyper-V Environment - Best Practices and Performance Recommendations

Running SQL Server with Hyper-V Dynamic Memory - Best Practices and Considerations

TEMPDB Capacity Planning and Concurrency Considerations for Index Create and Rebuild

The Data Loading Performance Guide 2008

Troubleshooting Performance Problems in SQL Server 2005

Troubleshooting Performance Problems in SQL Server 2008

Tuning the Performance of Change Data Capture in SQL Server 2008

Diagnosing and Resolving Latch Contention on SQL Server

Diagnosing and Resolving Spinlock Contention on SQL Server

 

Service Broker

Service Broker: Performance and Scalability Techniques

 

Full Text

SQL Server 2005 Full-Text Queries on Large Catalogs: Lessons Learned

 

Analysis Services

Accelerating Microsoft adCenter with Microsoft SQL Server 2008 Analysis Services

REAL PRACTICES: Performance Scaling Microsoft SQL Server 2008 Analysis Services at Microsoft adCenter

Resolving Common Connectivity Issues in SQL Server 2005 Analysis Services Connectivity Scenarios

Scale-Out Querying with Analysis Services

Scale-Out Querying with Analysis Services Using SAN Snapshots

The Analysis Services 2008 Performance Guide

Analysis Services 2005 Performance Guide

Analysis Services Distinct Count Optimization

Analysis Services Processing Best Practices

Analysis Services ROLAP for SQL Server Data Warehouses

Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

OLAP Design Best Practices for Analysis Services 2005

Precision Considerations for Analysis Services Users

Analysis Services Many-to-Many Dimensions: Query Performance Optimization Techniques

Scale-Out Querying for Analysis Services with Read-Only Databases

SQL Server 2008R2 Analysis Services Operations Guide

Analysis Services 2008 R2 Performance Guide

 

SQL Server Integration Services

Using SQL Server Integration Services and StreamInsight Together 

SSIS Tips Tricks and Best Practices: SSIS for Azure and Hybrid Data Movement

SSIS Tips Tricks and Best Practices: SSIS Operational and Tuning Guide

 

SQL General Administration

Database Upgrade to SQL Server 2008 Tools and Approaches

DBCC SHOWCONTIG Improvements in SQL Server 2005 and comparisons to SQL Server 2000

Disk Partition Alignment Best Practices for SQL Server

Enterprise Policy Management Framework with SQL Server 2008

 

 SQL Server Security and Audit

Reaching Compliance: SQL Server 2008 Compliance Guide

Manage Kerberos Authentication Issues in a Reporting Services Environment

 

SQL Server Application, Planning and Features

Best Practices for Migrating Non-Unicode Data Types to Unicode

Improving Microsoft CRM Performance and Securing Data with SQL Server 2008

New Spatial Features in SQL Server Code-Named “Denali” Community Technology Preview 3

Performance Optimizations for the XML Data Type in SQL Server 2005

SAP with Microsoft SQL Server 2005: Best Practices for High Availability, Performance, and Scalability

SQL Server 2005 Deployment Guidance for Web Hosting Environments

SQL Server 2008 with Siebel CRM Applications"

SQL Server 2008 with Siebel, PeopleSoft and JD Edwards Applications

SQL Server Consolidation Guidance

SQL Server Best Practices – Implementation of Database Object Schemas

ComparingTables Organized with Clustered Indexes versus Heaps

DataCompression: Strategy, Capacity Planning and Best Practices

FILESTREAMDesign and Implementation Considerations

Implementinga SQL Server Fast Track Data Warehouse

MicrosoftSQL Server 2005 Tuning Tips for PeopleSoft8.x 

The Impact of Changing Collations and of Changing Data Types from Non-Unicode to Unicode

Using SQL Server PowerPivot for Excel to Analyze SAP Data

XML Support in Microsoft SQL Server 2005

BizTalk Server 2009 Hyper-V Guide

SQL Server 2008 with Siebel CRM Applications

Working with the New SQL Server 2008 Collations and Earlier Versions of the Native Data Provider

Microsoft SQL Server 2008 R2 PowerPivot Planning and Deployment

New Spatial Features in SQL Server Code-Named “Denali” Community Technology Preview 1

Onboarding SQL Server Private Cloud Environment

 

SharePoint

Configure a Sharepoint Server Farm for minimal downtime during software updates

Configure disaster recovery across SharePoint farms by using SQL Server log shipping

Configuring Kerberos Authentication for Microsoft SharePoint 2010 Products

PowerPivot for SharePoint - Existing Farm Installation

PowerPivot for SharePoint - Transitioning from a Single Server to SharePoint Farm

PowerPivot for SharePoint: Single Server Installation

PowerPivot Technical Diagram: PowerPivot Client/Server Architecture

PowerPivot Technical Diagram: PowerPivot Security Architecture

Edgenet Realizes the Power of Dynamic IT and Self-Service BI

  

Reporting Services

Develop Reporting Services reports using Analysis Services data; a SQL Server 2008 technical case study

 

A lot of electronic ink has been spilled over our recent (and sudden) announcement that we are retiring the highest-level certifications here at Microsoft. At the same time, I'll be teaching a one-day class at the SQL Saturday event in Cambridge, UK on certifications. Are certifications worth the effort anymore, and should you attend the pre-con session?

Yes, and yes. I've gotten lots of certifications from Microsoft, Oracle, Sun, even Novell. I even took a class when I worked at NASA on circuitry connections - we still fixed computers back then by soldering and wiring (I'm old, don't make that face). Why did I take all these certifications? Well, in some cases it was to further my career - an employer put a "Must have an MCSE" on the job requirement. But most of the time a job requirement wasn't the reason . I believe to truly excel at your career you have to constantly learn and apply new information.

Certification certainly isn't required to learn - far from it. I've always been a self-motivated learner, but having the pressure of a timeline, a structured set of goals to hit, and an exam can be very powerful. I don't like to fail. So I studied, practiced, and learned, practiced tests, and memorized information. Did the certification make me a better professional? Absolutely. Is certification the only reason and method I used to study? Not by a long shot.

Now that I'm here at Microsoft, I have to learn metric ton-loads of new tech each week. It's overwhelming at times, so you have to learn to develop a process to quickly learn and apply new information - and that's why I put this pre-con together. I'll share with you the way I learn, and you'll get the benefit of others that I've learned from here at Microsoft on their learning styles and methods.

This is a workshop-style pre-conference session to learn not only how to get a Microsoft certification in SQL Server, but how to create your own study plans for any technical discipline, using SQL Server technical information to learn during the session. The certification information is very useful - but we'll do so much more. It's not a sit-down-and-listen session - you'll be doing lots of work, developing your own plans, and even moving around a bit. You'll learn the certifications that exist for SQL Server, from the simplest through the highest certification levels (including creating your own "MCSM" program), the resources you have for studying for them, such as books, links, videos and online courses. Using a hands-on workshop process, you'll walk through these materials and get training on both test-taking and technical topics that will prepare you to take your certifications to the level you're looking for. We'll focus on the process of learning, using technical topics dealing with SQL Server with real-world application of what you learn. You'll learn to prepare for a certification, along with invaluable technical details of the SQL Server platform.

 So if you're fortunate enough to be able to travel to the UK and the beautiful city of Cambridge, join the session. You'll learn one of the most important skills there is - learning to learn.

Some great work is being done by the Windows Server and System Center CAT related to Hybrid Cloud and… Wait for it… Provisioning SQL Server as a Service! 

A snippet from the recent blog post Provisioning SQL Server as a Service with the System Center stack;

“Can I use System Center to provision SQL Server components?”… “Can I provide database as a service to my business users or application owners?”…

The ability to automate SQL Server provisioning is probably one of the top customer requests I’ve heard about over the last 12 months!

The goal of this post is not only to confirm that System Center can indeed provide this capability, but also to provide a downloadable sample solution – the “SQL Server Self-Service Kit” - that you can reuse and adapt to your own needs.

 

SQL Server Self-Service Kit : Provisioning SQL Server as a Service with the System Center stack

http://blogs.technet.com/b/privatecloud/archive/2013/09/06/sql-server-self-service-kit-provisioning-sql-server-as-a-service-with-the-system-center-stack.aspx

 

Check out the Sample solution that includes 18 Orchestrator runbooks, 1 VMM service template, 1 Service Manager management pack and some unattended scripts, all geared towards SQL Server provisioning and self-service.1

 

SQL Server Self-Service Kit

http://gallery.technet.microsoft.com/SQL-Server-Self-Service-Kit-94c1e742

SQL Server Self-Service Kit : Gimme the details (part 1)

http://blogs.technet.com/b/privatecloud/archive/2013/09/06/sql-server-self-service-kit-gimme-the-details-part-1.aspx

SQL Server Self-Service Kit : Gimme the details (part 2)

http://blogs.technet.com/b/privatecloud/archive/2013/09/06/sql-server-self-service-kit-gimme-the-details-part-2.aspx

 

With the preview release of System Center 2012 R2, there are a few additions to VMM that make clustering with Windows Failover Clustering in guest VMs significantly easier. In this post, I’d like to highlight a few of those improvements as they relate to services.

Clustering Guests the Old Way

To be clear, Windows Failover Clustering doesn’t really work any differently on a physical host or a VM. As long as your VMs have connectivity to each other and access to shared storage, the process of clustering VMs isn’t any different. The big hurdle with a VM is shared storage.

Previously, it’s been possible to use iSCSI to provide shared storage. The general process required for this is as follows:

1. Provision iSCSI target and prepare storage
2. Configure target to allow connection from VMs
3. Deploy VMs
4. On all VMs, configure their initiators for the target
5. On one VM, online, partition, and format the drives
6. On one VM, create a cluster
7. Install any clustered applications

One might imagine that this can be a bit of a headache for repeated deployments of a clustered service. Doing this manually is prone to all sorts of error. It’s perfectly feasible to author a runbook to take care of the first two steps and deploy a service from a template for the remaining steps. Note that the scripts within the service template will need extra logic to discern which commands to run on which VM.

The New, Easier Way

The first new component to discuss is Shared VHDX, which is an easier way to share storage in VMs.

Shared VHDX

Virtual Machine Manager 2012 R2 provides a means to share a single VHDX file across a tier within a service. This means that when the VMs in that tier first boot, all the storage provisioning has been completed and the disk appears to all VMs without any additional configuration. At this point, the first VM can online, partition, and format the disk in preparation for its use as a clustered volume.

Note: The same disk access patterns apply for shared VHDX as for shared storage. That is, simultaneous writes can corrupt data and the disk access must be controlled by some clustering mechanism (for example, Windows Failover Clustering).

There are a few requirements for using shared VHDX within a service template. The service template must be marked as highly-available. The VMs in a highly-available service template can only be deployed to clustered hosts. Also, the disk cannot be an OS disk (sharing OS disks is never supported). It’s also worth mentioning that shared disks can be formatted, and contain data.

From within service templates in VMM, you’ll find the shared VHDX option within the Hardware Configuration of a machine tier’s properties. A VHDX that is attached to a SCSI adapter can be set to be shared amongst all VMs in that tier. After deployment, the shared disk will appear in all VMs as attached storage just as an additional VHDX should. From here, cluster configuration can begin.

When deploying multiple VHDX images to a machine for clustering (for example a witness disk and a data disk), the same disk image cannot be used within the library. For each VHDX image deployed to the same tier, you’ll need a separate VHDX file in the library. For example, if each VM in a tier had two VHDX disk images that are shared across the tier and an OS image, a total of three disk images will be required in the library (one with the Sysprep’d OS and the other two for cluster).

To simplify some of the logic in cluster configuration, VMM 2012 R2 has provided additional script hooks that can assist with cluster deployment logic. I’ll discuss the four new hooks next.

New Script Hooks

When clustering, there’s usually a need for asymmetric provisioning: treating the first node differently from the remainder of the nodes. Before, this required complicated script logic. Now, we can simplify efforts by using the new script hooks provided in VMM 2012 R2.

The first new script hook is Creation: First VM. When the first VM in a service tier is created, the specified script will run on this VM and only this VM. All of the operations that are performed once (disk formatting, cluster creation), can be included in this script. Now, subsequent VMs can make simple assumptions about the current state of the environment and simply join the existing cluster.

For those other VMs, there’s a script hook for those scripts, which is Creation: VMs After First. This script will run on other VMs created and not the first VM.

Now, this fixes the problems with creation and with scaling out, but what about scaling in? Two other hooks were added for this purpose. When scripts are marked Deletion: VMs Before Last, all but the last VM run this script. That is, if you decide to scale-in a tier in a service from two VMs to a single VM, this script will run on the second VM prior to its deletion. Any actions such as removing a node from a cluster can be performed during this operation.

Likewise, a script marked as Deletion: Last VM will run on the last VM when the service is scaled to zero for whatever reason. Using this hook, scripts can perform clean-up operations so that the corresponding Creation: First VM scripts will run as if in a clean environment.

With a combination of shared VHDX and the new script hooks, deploying clustered services is easy.

Obligatory Example

Explanation and conjecture is nice, but nothing speaks for new functionality like a cold, hard demo. As such, I’ve put together a simple example of a clustered service. In this example, we’ll have a two-node guest cluster running SQL installed as a failover cluster instance.

Process

First, we should outline the process of deploying a SQL Failover Cluster Instance on physical machines, and then we can convert it to template logic. The process is as follows

• Configure first server
• Install OS
• Setup roles and features
• Failover Clustering
• Failover Cluster Management Tools
• Failover Clustering Module for Windows PowerShell
• Join server to a domain
• Create a new Windows Failover Cluster
• Partition and format disks in cluster
• Run the SQL installation as a first node in a new failover cluster
• Configure second server (and any subsequent servers)
• Install OS
• Setup roles and features
• Failover Clustering
• Failover Cluster Management Tools
• Failover Clustering Module for Windows PowerShell
• Join server to a domain
• Join existing Windows Failover Cluster
• Run the SQL installation as a new node in an existing failover cluster

One might notice that this fits very well within the confines of the new script hooks. The only steps that differ are italicized and the machines are otherwise identical. To convert this to a service template, we can use a single tier service and take care to specify which scripts run when.

Scripts

Clustering on the First Node

We’ll need two clustering scripts: one to create a cluster and one to join an existing cluster. To create the cluster, we can use the New-Cluster cmdlet. After the cluster has been created, disks may be added to the cluster. This can be done by getting the disks available to the cluster with the Get-ClusterAvailableDisk cmdlet and then piping that into the Add-ClusterDisk cmdlet.

There is one caveat to mention here: cluster disk ordering. Disks don’t necessarily appear to the operating system in the same order as they appear to a Windows Failover Cluster. That is, the PowerShell cmdlet Get-ClusterAvailableDisk may return a disk number 5 as ‘Cluster Disk 1’ and disk number 1 as ‘Cluster Disk 2’. Now, the New-Partition cmdlet expects the drive number as it appears to the operating system (as in the Get-Disk cmdlet). So, if the drives are formatted sequentially, this may correspond to ‘Cluster Disk 1’ being G: and ‘Cluster Disk 2’ being D:, which provides some complication to specifying cluster disk name and drive letter during a SQL installation.

As a possible alternative, consider first bringing online and partitioning the disks without formatting them (this will allow the disks to be detected by the cluster). Next, run the Get-ClusterAvailableDisk cmdlet to get a set of Name/Number pairs. Now, use the Set-Partition cmdlet to assign drive letters. Lastly, you can use the Format-Volume cmdlet, specifying the driver letters. This will provide you with a deterministic ordering to the cluster disk name and driver letter pairs. A script example below should make this clear.

Next, the cluster quorum settings can be set via Set-ClusterQuorum and cluster validation can be run with Test-Cluster. Note that, by default, SQL installation will not proceed on an a cluster that has not been validated. Putting this all together, the cluster script for the first host may look something like this:

Param($clustername,$clusterIP);

functionTime {
    return[System.DateTime]::Now.ToString("yyyy.MM.dd hh:mm:ss");
}

functionLog($out) {
    $out=[System.DateTime]::Now.ToString("yyyy.MM.dd hh:mm:ss") +" ---- "+$out;
    Write-Output"$out"
}

if($IP) {
    Log"Creating cluster $clustername on $env:COMPUTERNAME with IP $clusterIP";
    New-Cluster-Name$clustername-Nodelocalhost-StaticAddress$clusterIP|Out-Null;
} else {
    Log"Creating cluster $clustername on $env:COMPUTERNAME using DHCP";
    New-Cluster-Name$clustername-Nodelocalhost|Out-Null;
}

Log"Getting available disks...";
$disks=Get-Disk|where {$_.FriendlyName -eq"Microsoft Virtual Disk"};

foreach($volin$disks) {
    $num=$vol.Number;
    Log"Starting with drive $num";
    Log"Bringing drive online...";
    Set-Disk-Number$num-IsOffline$False;
    Log"Initializing partition table as MBR...";
    Initialize-Disk-Number$num-PartitionStyleMBR;
    Log"Creating single partition...";
    New-Partition$num-UseMaximumSize|Out-Null;
    Log"Done with drive $num";
}

$clusterdisks=Get-ClusterAvailableDisk|SelectName,Number;
Log"Formatting disks and assigning drive letters...";
$letters=69..90|%{[char]$_};
foreach($diskin$clusterdisks) {
    $num=[int]($disk.Name -replace'Cluster Disk ','');
    $letter=$letters[$num];
    $drivenum=$disk.Number;
    Log"Formatting Cluster Disk $num (Drive $drivenum) and assigning drive letter letter...";
    Set-Partition$disk.Number -PartitionNumber1-NewDriveLetter$letter|Out-Null;
    Format-Volume-DriveLetter$letter-FileSystemNTFS-Confirm:$false|Out-Null;
}

Log"Adding disks to cluster...";
Get-ClusterAvailableDisk|Add-ClusterDisk;
Log"Assigning witness disk";
Set-ClusterQuorum-NodeAndDiskMajority'Cluster Disk 1';
Log"Validating cluster...";

$results=Test-Cluster-WarningActionSilentlyContinue;
if(Select-String'

'$results) {
    Log"Cluster Validation errors...";
    Write-Error"Validation failure. See '$results' for details.";
    exit-1;
}
Log"Validating cluster complete";

 

The Log function in this script is useful for standardizing log output for troubleshooting template deployment failures. Note that an argument to this script is $clusterIP, which allows for deployment in a static IP environment. VMM is not able to allocate additional IP addresses from a static IP pool for objects like cluster names and SQL cluster names, so these can be supplied as service settings in a service template. At deployment, the static IP can either be manually assigned from the static IP pool in VMM with cmdlets, or pre-allocated and deployed with a runbook. Although it is possible for this script to contain the logic to request an IP address from the VMM server, the script would need the name of the VMM server and there would need to be connectivity to the VMM server from within the VM.

Clustering on the Other Nodes

This is very straightforward. The only thing the other nodes need to do is join an existing cluster. Piping Get-Cluster into Add-ClusterNode is only a single line of code. Very simple, generally. There may be some other considerations, however. For example, DNS propagation delay may cause hiccups to this deployment. You might work around this by adding a block to wait on DNS, and then subsequently wait on the cluster name, for whatever reason. You’ll want to make sure that in doing this, however, you avoid two things: waiting forever or wasting CPU cycles. The solution is to set an appropriate timeout for the script and sleep in between checks. Even adding this, the script is still fairly small:

Param($clustername);

functionTime {
    return[System.DateTime]::Now.ToString("yyyy.MM.dd hh:mm:ss");
}

functionLog($out) {
    $out=[System.DateTime]::Now.ToString("yyyy.MM.dd hh:mm:ss") +" ---- "+$out;
    Write-Output "$out";
}

while(!$([System.Net.Dns]::GetHostAddresses($clusterName))) {
    Log"Waiting for DNS...";
    [System.Threading.Thread]::Sleep(20000);
}

Log"Adding localhost to $clustername";
Get-Cluster-Name$clustername|Add-ClusterNode-Namelocalhost;

Log"Validating cluster...";

$results=Test-Cluster-WarningActionSilentlyContinue;
if(Select-String'

'$results) {
    Log"Cluster Validation errors...";
    Write-Error"Validation failure. See '$results' for details.";
    exit-1;
}
Log"Validating cluster complete";

In some environments, you might consider pre-creating the DNS entries to avoid long delays. Since the drives have already been brought online and formatted, no further action is necessary on the other nodes. Note that the first node has already finished its complete deployment (including the SQL installation) prior to the other nodes running any of their scripts.

SQL Installation Configuration Preparation

Be sure to checkout Bruno Saille’s post for more guidance on provisioning SQL as a service in a System Center stack. I’ll just be covering a simple example as it relates to running SQL in a Windows Failover Cluster. As we will need to be running the SQL installation entirely without user interaction, you might want to read this article about commandline installation of SQL. There are many options and you’ll want to customize your deployment for your environment.

One way to easy the pain of all these options, especially if you’re more familiar with installing SQL with a GUI, is to work your way through the GUI install and then cancel before actually installing. When the SQL installer prepares an installation, it will save a settings file for the configuration and supply its location. I’ll do a quick walkthrough of generating a settings file.

First, start begin the installation as if creating a new failover cluster, and work through all the steps as usual (you’ll need an existing cluster to pass all the checks to generate the configuration).

Now, on the final page of the installer before beginning installation, take the given file that is referenced and use that as your configuration. Simple, and painless.

You’ll do the same process for the add node configuration, and everything should be done.

Doing things this way is probably the easiest way to generate a base configuration you can work with.

SQL Installation on the First Node

As the configuration file is already generated, it’s a simple matter of referencing that file when starting the install and providing any additional parameters (user names, passwords, IP addresses). For ease of deployment, you might consider two options for your configuration files.

The first option is to have a partial configuration file INI with all the static components saved within the custom resource. When running the install, this file could be referenced and additional options can be supplied as parameters to the installer.

The second option is to generate the entire configuration from within the script. This may be the most flexible of options, but requires a bit of work to ensure that everything works as expected.

You might come up with other ways to accomplish the task, as well. The key is that all the SQL installation components be on the system during deployment. The easiest way might be to add the SQL installation directory to the OS disk image, and perhaps delete that directory as the last deployment step, if desired. You might also consider pointing to UNC paths for some deployments. The details will, of course, depend on your environment. For this example, I chose to add the SQL installation directory to the OS disk image so I can reference it easily within deployment scripts.

My deployment is using DHCP and a very simple configuration, so I don’t need any special scripts at this point. My INI file looks like this:

[OPTIONS]
ACTION="InstallFailoverCluster"
ENU="True"
QUIET="True"
UpdateEnabled="False"
FEATURES=SQLENGINE,REPLICATION,FULLTEXT,DQ
UpdateSource="MU"
HELP="False"
INDICATEPROGRESS="False"
X86="False"
INSTALLSHAREDDIR="C:\Program Files\Microsoft SQL Server"
INSTALLSHAREDWOWDIR="C:\Program Files (x86)\Microsoft SQL Server"
INSTANCENAME="DEMO"
SQMREPORTING="False"
INSTANCEID="DEMO"
ERRORREPORTING="False"
INSTANCEDIR="C:\Program Files\Microsoft SQL Server"
FAILOVERCLUSTERDISKS="Cluster Disk 2"
FAILOVERCLUSTERGROUP="SQL Server (DEMO)"
FAILOVERCLUSTERIPADDRESSES="IPv4;DHCP;ClusterNetwork1"
FAILOVERCLUSTERNETWORKNAME="SQLCLUSTERDEMO"
COMMFABRICPORT="0"
COMMFABRICNETWORKLEVEL="0"
COMMFABRICENCRYPTION="0"
MATRIXCMBRICKCOMMPORT="0"
FILESTREAMLEVEL="0"
SQLCOLLATION="SQL_Latin1_General_CP1_CI_AS"
INSTALLSQLDATADIR="F:"
SQLUSERDBLOGDIR="F:\MSSQL11.DEMO\MSSQL\Data"
SQLTEMPDBDIR="F:\MSSQL11.DEMO\MSSQL\Data"
SQLTEMPDBLOGDIR="F:\MSSQL11.DEMO\MSSQL\Data"
FTSVCACCOUNT="NT Service\MSSQLFDLauncher$DEMO"

With this, I can simply run

C:\SQL\Setup.exe /ConfigurationFile=first.ini /IAcceptSQLServerLicenseTerms /AGTSVCACCOUNT=@Agent Account@ /AGTSVCPASSWORD=@Agent Password@ /SQLSVCACCOUNT=@SQL Account@ /SQLSVCPASSWORD=@SQL Password@ /SQLSYSADMINACCOUNTS=@SQL Admins@

Now, each of those can be service settings supplied at deployment time. You can see that this is easily customizable for your deployment environment.

SQL Installation on the Other Nodes

The process is similar for other nodes, but the generated configuration INI is a bit different, as the action is different and fewer inputs are needed:

[OPTIONS]
ACTION="AddNode"
ENU="True"
QUIET="True"
UpdateEnabled="False"
UpdateSource="MU"
HELP="False"
INDICATEPROGRESS="False"
X86="False"
INSTANCENAME="DEMO"
SQMREPORTING="False"
FAILOVERCLUSTERGROUP="SQL Server (DEMO)"
CONFIRMIPDEPENDENCYCHANGE="False"
FAILOVERCLUSTERIPADDRESSES="IPv4;DHCP;ClusterNetwork1"
FAILOVERCLUSTERNETWORKNAME="SQLCLUSTERDEMO"
FTSVCACCOUNT="NT Service\MSSQLFDLauncher$DEMO"

We refer to rest.ini instead of first.ini, and the SQL installation is otherwise the same.

Other Considerations

By default, a FCI install of SQL is not allowed on a cluster that has not been validated. Whenever significant cluster configuration changes occur, the cluster validation must run again. As such, cluster validation is run with the addition of each node. However, cluster validation can only be conducted on one node at a time.

More than just this limitation of cluster validation, it’s also important to note that SQL’s AddNode is not supported in parallel. If you write in complicated script logic to handle the cluster validation limitation, you should also handle running the SQL installs in a serial manner.

Because of these two points, and the fact that a tier with three nodes will deploy the second and third nodes in parallel, deployment is almost guaranteed to fail for more than two nodes. As a result, the tier should only be configured to deploy a default of one or two instances. After two instances have been deployed, the tier can be scaled out, one node at a time.

In some environments, a runbook offers great advantages. The runbook can obtain static IP addresses from VMM, pass them as parameters to the template service settings deployment, deploy the first two nodes, and then scale to any number of nodes in the tier in a serial manner.

In other environments, you might consider only using two VMs and scaling in/out as dictated by performance monitoring.

Conclusion

With the new additions in VMM 2012 R2, clustering applications within the guest has been greatly simplified. In this post, I’ve done my best to get you enough information to get started. Note that configuring a SQL cluster as a single tier has implications for simplifying more robust service templates, as well. Templates that include a web application with a SQL backend can now be deployed with a clustered SQL instance that is scalable, independent of the web frontend.

So I was going through the features in SQL Server 2014, seeing how and where they fit into the canon of SQL Server’s T-SQL. I’m going to add some of this for my advanced T-SQL Immersion Event at the end of September. SQL Server 2014 is supposed to be a release with no T-SQL changes at all. Not windowing enhancements, not time interval data types, nothing. So the effect on the T-SQL programmer should be minimal, right? Well, maybe not minimal.

The most visible new feature in SQL Server 2014 is known as Hekaton. In a perfect world, Hekaton tables would simply be a deployment decision by the DBA. The miracle “go faster” switch, just buy some more memory move your tables over. But it turns out that the surface area of tables and statements on tables in wide and deep. And in “V1″ of memory-optimized tables there are a number of constructs that they don’t support. These include some data types, as well as constraints and triggers. So T-SQL developers need to “make up” for lack of these by coming up with their own scheme, no mean feat although there are some examples of this type of code in the Books Online. And that’s just the beginning, the list of unsupported items means that the move of a table to memory-optimized won’t be a simple change to DDL.

As another example, the change from pages to hash buckets in the in-memory engine gives new credibility (well, gives credibility) to the concept of uniqueidentifier as primary key. With hash buckets, spreading out the key values won’t cause fragmentation, it should actually be as fast or faster. It is still a bigger data type though, which means more to keep in memory. Just buy a bit more memory and a fast disk for the log.

T-SQL compiled stored procedures that calls table-specific code is needed to make memory-optimized tables run as fast as possible. In fact, compiled stored procedures can only access memory-optimized tables. These aren’t your traditional stored procedures. They must be defined with a transaction isolation level, as well as other options. And these stored procedures also have a number of limitations, including limiting usage of some built-in functions, tempdb, and collation choice. They also compile on first use (when you create them initially and when you bring up SQL Server) and don’t recompile because of statistics changes. Finally, these procedures need error handling because, depending on isolation level, they might rollback due to the multi-version engine. Ah, the multi-version engine.. That may affect T-SQL programmers a little.

There’s more than just rowstore and linked list memory-optimized tables. This release can use columnstore (xVelocity) as the primary storage vehicle, rather than as an additional copy of your warehouse data as in SQL Server 2012. And the columnstore is updatable now. And because columnstore is “V2″ now, programmers can “undo” almost every workaround that was needed to take full advantage of “Batch Mode” processing because constructs like outer joins will now use Batch Mode automatically.

Let’s not forget the new cardinality estimation. If you’re fearing plan regressions, you can always opt out, but sounds like some tuning, as well as better estimates leading to better plans, will be going on. And there’s always parallelism in SELECT..INTO too.

So ever if they never add another T-SQL feature to SQL Server 2014 by RTM, there’s lots to talk about that pertains to the way you’ll write T-SQL in future as well as, of course, lots about how we write it for “best results” today. See you later this month.

@bobbeauch

The post Integrating SQL Server 2014 constructs into the T-SQL canon appeared first on Bob Beauchemin.

Last week I was checking on some stuff and noticed that a couple databases were in simple mode – unusual, but not necessarily wrong, so I asked someone on the team that manages the environment about it. He said that because of the usage it wasn’t critical but they did prefer them to be set to full recovery, and then he made the change – something they have the flexibility to do without a change ticket on small changes.

Easy stuff, right?

Twenty or so minutes later I saw an alert about log backups failing. I opened it up and right away knew what had happened – he didn’t follow up the change by running a full backup, which meant the log backups for that database would continue to fail until the nightly full backup ran. In the scheme of things this is a tiny mistake. All the other logs were backed up as usual, the database was no more at risk than it had been, and the error would eventually self correct.

Calling it a mistake is probably drama, maybe better to say a glitch. Still, a not-too-painful reminder that I should have mentioned the need for a full backup, and a reminder that in the work we do even the simplest change can have an unexpected impact.

wabrandsma writes "Quoting Bruce Schneier in the Guardian: 'The NSA has undermined a fundamental social contract. We engineers built the internet – and now we have to fix it. Government and industry have betrayed the internet, and us. This is not the internet the world needs, or the internet its creators envisioned. We need to take it back. And by we, I mean the engineering community. Yes, this is primarily a political problem, a policy matter that requires political intervention. But this is also an engineering problem, and there are several things engineers can – and should – do."

Read more of this story at Slashdot.

CowboyRobot writes "A Tripwire survey of 1,320 IT personnel from the U.S. and U.K. showed that most staff 'don't communicate security risk with senior executives or only communicate when a serious security risk is revealed.' The reason is that staff have resigned themselves to staying mum due to an environment in which 'collaboration between security risk management and business is poor, nonexistent or adversarial,' or at best, just isn't effective at getting risk concerns up to senior management."

Read more of this story at Slashdot.

The following is a useful summary table that outlines the physical memory limits found with Microsoft Windows Server 2012 (W2K12).  Just as a reminder Windows Server 2012 is only available as an X64 distribution.

Although I’m not a big fan of the new user interface I will be starting to move my virtualization lab based servers over to W2K12 sometime soon.

Windows Server 2012 Version	Physical Memory Limit
Foundation	32GB
Essentials	64GB
Standard	4TB
Datacenter	4TB
Storage Server 2012 Workgroup	32GB
Storage Server 2012 Standard	4TB
Hyper-V Server 2012	4TB

The post Microsoft Windows Server 2012 Memory Limits appeared first on TechHead and was written by Simon Seagrave.

Today a coworker asked me about extending an EagerZeroedThick disk, as they had a customer inform them that extending an EagerZeroedThick disk actually turned it LazyZeroed. Although this is somewhat true, it is not 100% accurate. What actually happens is the extension becomes LazyZeroed but the original portion of the disk still remains EagerZeroedThick. This is actually by design, if you want to extend the VMDK and ensure that it is all EagerZeroedThick you must specify a flag in the vmkfstools command to do so. Extending an EagerZeroedThick VMDK # vmkfstools -X 10G -d eagerzeroedthick /path/to/vmdkfile.vmdk In the above syntax example it is critical that you specify eagerzeroedthick as the disk format.

VMware has announced a new level of certification for aspiring VMware administrators, users or tech junkies.  The Associate level (VCA) is now the introductory certification level for DataCenter Virtualization, Cloud, Network Virtualization and End-User Computing.  One big difference between Associate and Professional is that an Associate level certification does not require training in order to receive certification once passing the exam.  Although the new Associate certification is a step below Professional, you’ll still receive a VCA logo, have access to an exclusive VCA portal and logo merchandise store, be invited to beta exams and classes as well as receive discounted admission to VMware events such as VMworld. VMware has also announced a handful of new certification paths.  I’ve listed the new certification paths below.  Please note that I did not include version 4 certification paths not because they’re not important but to streamline the content.  You can find information on all of the below certifications at the following link. VMware Certification CLOUD VMware Certified Associate – Cloud (VCA-Cloud) NEW VMware Certified Professional – Cloud (VCP-Cloud) VMware Certified Advanced Professional – Cloud Infrastructure Design (VCAP-CID) VMWare Certified Advanced Professional – Cloud Infrastructure Administration (VCAP-CIA) VMware Certified Advanced Professional – Cloud Governance Coming Soon VMware Certified Design Expert – Cloud (VCDX-Cloud) NEW DATA CENTER VIRTUALIZATION VMware Certified Associate – Data Center Virtualization (VCA-DCV) NEW VMware Certified Professional – Data Center Virtualization (VCP-DCV) VMware Certified Advanced Professional – Data Center Administration (VCAP-DCA) VMware Certified Advanced Professional – Data Center Design (VCAP-DCD) VMware Certified Design Expert – Data Center Virtualization (VCDX5-DCV) END USER COMPUTING VMware Certified Associate – Workforce Mobility (VCA-WM) NEW VMware Certified Professional 5 – Desktop (VCP5-DT) VMware Certified Advanced Professional – Desktop Administration (VCAP-DTA) Coming Soon VMware Certified Advanced Professional – Desktop Design (VCAP-DTD) VMware Certified Design Expert – Desktop (VDCX-DT) Coming Soon NETWORK VIRTUALIZATION VMware Certified Associate – Network Virtualization (VCA-NV) Coming Soon CLOUD APPLICATION PLATFORM Certified Spring Professional Certified Spring Web Application Developer Certified Spring Enterprise Integration Specialist

When you have a synchronous Mirrorview setup for the VNX or Clariion, you like to be able to perform some Disaster Recovery testing once in a while. When you have many replicated LUN’s, scripting can speed up some things for you regarding the Snapview configuration steps. Inside this post, I will show a Powershell script I created for automating some steps on the storage array resulting in presenting the snapped LUN’s to a VMware vSphere environment for example. First of all, make sure you are logged on to a host that has an IP connection to both SP’s of the Clariion / VNX and besides this, install the latest version of the Naviseccli. Now, create a CSV file which summarize the LUN ID’s you want to make part of the Disaster Recovery test. [plain]LUN 700 701 702 703[/plain] Now you can use the following Powershell script for automating the following steps: Create Naviseccli security file Create Snapshots Add Snapshots to the correct Storage Group Create and start a consistent Snapview session Active the LUN’s to the Snapview session [powershell] #This script has the following functions based on CSV with the LUN ID’s summarized # Create security files # Create snapshots # Add snapshots to Storage Group # Start Snapview session # Activate snapshots on session #Specify the information down here $csv = Import-Csv ‘D:\Scripting\Powershell\Snapview\LUN.csv’ $SPA = ’x.x.x.x’ $SPB = ‘x.x.x.x’ $User = ‘user’ $Password = ‘password’ $Scope = ’0′ $SG = ‘StorageGroup’ $SnapSession = ‘SnapSession’ $SnapPrefix = ‘Snap_’ $StartHLU = 100 #!!!!!Do not edit from here!!!!!! $a = [char]34 $String = $a #Add the user security file in case it is not present $UserSecurity1 = (‘naviseccli -h ‘ + $SPA + ‘ -addusersecurity -user ‘ + $User + ‘ -password ‘ + $Password + ‘ -scope ‘ + $Scope) Write-Host $UserSecurity1 invoke-expression $UserSecurity1 $UserSecurity2 = (‘naviseccli -h ‘ + $SPB + ‘ -addusersecurity -user ‘ + $User + ‘ -password ‘ + $Password + ‘ -scope ‘ + $Scope) Write-Host $UserSecurity2 invoke-expression $UserSecurity2 #For every LUN in csv we perform some actions $HLU = $StartHLU foreach ($line in $csv) { #Create Snapshot $LUN = $($line.LUN) $SnapName = $SnapPrefix + $LUN $CreateSnap = (‘naviseccli -h ‘ + $SPA + ‘ snapview -createsnapshot ‘ + $LUN + ‘ -snapshotname ‘ + $SnapName) Write-Host $CreateSnap Invoke-Expression $CreateSnap #Add Snapshot to the Storage Group $AddToSG = (‘naviseccli -h ‘ + $SPA + ‘ storagegroup -addsnapshot -gname ‘ + $SG + ‘ -hlu ‘ + $HLU + ‘ -snapshotname ‘ + $SnapName) Write-Host $AddToSG Invoke-Expression $AddToSG $HLU = $HLU + 1 $String = $String + $SnapName + ‘,’ } #Start the Snapview session with the specified snapshots in a consistent mode $string = $string.Substring(0,$string.Length-1) $String = $String + $a Write-Host (‘naviseccli -h ‘ + $SPA + ‘ snapview -startsession ‘ + $SnapSession + ‘ -snapshotname ‘ + $String + ‘ -consistent’) Invoke-Expression (‘naviseccli -h ‘ + $SPA + ‘ snapview -startsession ‘ + $SnapSession + ‘ -snapshotname ‘ + $String + ‘ -consistent’) #The last step is to activate all snapshots foreach ($line in $csv) { $LUN = $($line.LUN) $SnapName = $SnapPrefix + $LUN $Activate = (‘naviseccli -h ‘ + $SPA + ‘ snapview -activatesnapshot ‘ + $SnapSession + ‘ -snapshotname ‘ + $SnapName) Write-Host $Activate Invoke-Expression $Activate }[/powershell] Now specify the following stuff in the script: Location of the CSV file IP addresses of both SP’s Username, password and scope to use Name of the Storage Group for the to be created snapshots Name of the Snapview session Desired prefix for the snapshot names First to be used Host LUN ID in the desired Storage Group When running this script, from the VMware vSphere environment you can now start adding the VMFS datastores to the ESX hosts and add VM’s to the inventory for testing purpose. Do not forget to configure the required amount of Reserved LUN space based on EMC best practices.

My last post discussed the issues with EMC’s strategy around the marketing of their new VNX platform.  Some people will think picking holes in a vendor’s strategy is easy writing, so for those of that persuasion, I’ve taken the time to look at EMC’s portfolio and express an opinion on where EMC should go next. The Portfolio As I’ve discussed, EMC has a wide range of storage array technology now. VMAX – branded as Symmetrix then DMX, this is the original storage platform that made the company big.  Twenty years ago when storage was widely distributed in servers, Symmetrix brought a new way of working with consolidated storage and advanced features like RAID and cache.  VMAX has done well over the years and still is a major part of EMC’s portfolio, but some of the technology features are looking a little old, with many layers of software being used to implement advanced features such as thin provisioning. VNX – this platform started life as the CLARiiON, when Data General was acquired by EMC in 1999.  At the time, the CLARiiON was an advanced piece of technology, with dual controllers and RAID management functions.  The platform was merged with the Celerra NAS gateway to produce VNX and up until VNX2 that heritage could still be seen in configuration dumps. Atmos – EMC’s attempt at scale-out low cost cloud storage.  The platform had no frills (like RAID), which perhaps scared customers who were used to the traditional VMAX storage model.  I’m not sure how successful Atmos has been but I hear very little about it. Isilon – Scale-out NAS.  Acquired by EMC in 2010, this is a good example of future storage architectures using scale-out functionality XtremIO – Not officially a GA product (but almost there), XtremIO is EMC’s attempt at scale out all-flash.  It is yet another acquisition, for a rumoured $430m. ScaleIO – an even more recent acquisition (July 2013), ScaleIO is a distributed scale-out multi-node solution that utiilises existing servers and storage resources. Project Nile – the latest EMC teaser – a block, NAS and object scale-out solution that can be purchased “on demand” and ships to the customer fully configured. So what do we have here?  Well, the legacy VMAX and VNX platforms are monolithic and dual architecture designs.  These architectures don’t lend themselves well to the development of cloud and server scale-out architectures that we can expect to see in the future.  In addition, they were designed when storage needed to be centralised for resiliency and performance.  However, these days hard drives and server components are significantly more reliable than ever before and flash storage is delivering performance with high I/O density (IOPS per GB).  That’s why scale out architectures such as SolidFire and Nutanix have been successful and why with Open Computing architectures we are moving to the server as the level of granularity. In a recent post, Chad Sakac discusses the scale-out issue, but has to contradict himself in order to justify the existence of VNX2.  Check out the paragraph and statement “scale-out is the future, scale-up is the past”.  Unfortunately for Chad, as a marketer he has to toe the line on VNX because of the huge mass of existing customers who still believe in the VNX platform. Recycle Reuse The clear message from EMC’s portfolio is that scale-out will be the future.  The question is how to get there.  Well, EMC have all the key pieces; Atmos provides the raw storage capability; XtremIO, Isilon and ScaleIO provide the architecture.  In fact, if the XtremIO solution is so good, then surely it could be retro-fitted with hard drives to deliver a hybrid solution.  Let’s think about that one for a moment.  If you look at storage usage today, you will typically find that on any large shared storage array, 80% of the activity is being done by 20% of the data, with a long tail of small volume I/O.  I talked about this three years ago in a post on Wide Striping. So, with that in mind, an array could be constructed using the XtremIO architecture with just a few SSDs and the majority capacity provided by large slower hard drives.  The question is whether the architecture can target writes effectively.  I’ve talked about that too, in this 2009 post on automated tiering. Using the XtremIO architecture as the basis of a hybrid array could work, or alternatively, taking the ScaleIO software and using other EMC hardware components could deliver a solution as well. The Architect’s View If EMC have all the pieces to evolve VNX, then why have they not done so?  There are probably a number of reasons: XtremIO and ScaleIO are very early technology solutions and so not fully featured with things like snapshots, replication, clones, thin provisioning and so on.  There would be significant development work required to get this into place and these are only recent acquisitions. VNX is a cash cow.  EMC don’t want to kill off a platform that could keep revenues coming in for many years yet. Upsetting the ecosystem.  Many organisations will have built processes and skills around VNX.  Radical change to the platform would represent a major disruptive force. The problem EMC have is that other vendors, have already made the move.  NetApp (aka “Brand N”) have re-engineered Data ONTAP from the Spinnaker platform to take over from legacy ONTAP, the so-called 7-mode and cluster-mode versions.  Although these were totally different operating systems, cluster-mode has been changed to support all of the previous 7-mode features.  The only issue for customers will be migration.  NetApp are hoping 7-mode will fade away gracefully as customers transition themselves over.  HP have let the EVA die off in preference for the 3PAR platform, which now scales from entry level to enterprise and includes all-flash. It could well be that EMC have known these issues all along and VNX2 is merely a stop-gap to the release of a new platform that solves all these problems.  Could this be Project Nile? We will have to wait and see.   Related Links Speed2lead – Part1 – Next Generation VNX (virtualgeek blog) Enterprise Computing: The Benefits of Wide Striping – Avoiding a Long Tail Enterprise Computing: Automated Tiering – Why Move The Data?   Comments are always welcome; please indicate if you work for a vendor as it’s only fair.  If you have any related links of interest, please feel free to add them as a comment for consideration. Subscribe to the newsletter! – simply follow this link and enter your basic details (email addresses not shared with any other site). Copyright (c) 2013 – Brookend Ltd, first published on http://architecting.it, do not reproduce without permission.

In my previous post on the Next-Generation VNX, I spoke about some of the highlights of the new platform. In this post I’d like to dive a little deeper into the deduplication feature, because I think this stuff is pretty neat. In the interests of transparency, I’m taking a lot of this information from briefings I’ve received from EMC. I haven’t yet had the chance to test this for myself, so, as always, your mileage might vary.

One of the key benefits of deduplication is reduced footprint. Here’s a marketing picture that expresses, via a simple graph, how deduplication can help you do more, with less.

There are 3 basic steps to deduplication:

1. Discover / Digest;
2. Sort / Identify; and
3. Map / Eliminate.

The Discovery phase is basically generating hashes of 8KB blocks using unique digests. It’s then sorted to identify chunk candidates for deduplication. The duplicates are then mapped and the space is freed up. Digests are used as pointers for the unique data chunks.

Deduplication can be turned on or off at the LUN level. Pools can contain both deduplicated and “normal” LUNs. Also, note that the total number of LUNs on a system can be deduplicated – there is no separate limit applied to the deduplication technology.

The deduplication properties of a LUN are as follows:

• Feature State – On or Paused
• State – Off, Enabling, On, Disabling
• Status – indicates any problems during enabling or disabling

The deduplication properties of a Pool are as follows:

• State – Idle (no deduplicated LUNs), Pending (between passes), Running (currently running a dedupe pass) and Paused (pool is paused)
• Deduplication Rate – High, Medium, Low (Medium is current default)

Note that dedplication can be paused at a system level for all pools.

When deduplication is turned off on a LUN it is migrated out of the deduplication container within the pool. 8 simultaneous migrations per system can occur, and it obviously reduces the consumed space in the deduplication container.

At a high level, deduplication interoperability is there:

• Works with FAST VP (dedupe LUNs behave as a single entity, when dedupe is turned off it goes back to per-LUN FAST VP settings)
• Supports Snaps and Clones (VNX Snapshots are lost when enabling or disabling, Reserved LUNs for SnapView Snaps cannot be deduplicated)
• Support for RP, MV and SAN Copy
• LUN migration works, although moving between pools means deduplication is lost as it’s pool-based
• Compression is not supported.

And that’s fixed-block deduplication for the next-generation VNX in a nutshell. When I get my hands on one of these I’ll be running it through some more realistic testing and scenarios.

Obvious point: I work at EMC.

Less obvious point: EMC is making use of the hashtag #NotAppy as you may have seen.  Not everyone is finding humor in it:

The whole #NotAppy marketing ploy from @EMCcorp seems sad and tacky. Why punch downwards when you’re on top?

— Justin Hurst (@Tomorrowisms) August 29, 2013

I want to take a moment and dig into how I feel about these from a factual and a speculative standpoint.

FACT - we at EMC have released a number of photos, and even a movie, calling NetApp into question.

FACT –  NetApp is part of the Fortune 500 and is deployed in many others companies that make it on the list.

FACT – many community contributors in the partner space, and in the EMC Elect, sell NetApp to their customers.

OPINION – A level of quality is required to be deployed in our competitive market.

Said another way, no company in production as tier 1 storage is without respectable qualities universally recognized by experts in our industry.

Data integrity is a no-compromise conversation in the technical industry. Checkboxes like high availability and strong performance results in standards such as SPC-2 benchmarks are required.

OPINION - I respect the technical skill running through every member of the EMC Elect, which translates into a trust that I can have some respect for NetApp technology.

OPINION - I see NetApp as our hometown rival. We shout and wave our arms in public. The crowd goes wild.

Then we all go out together for a beer.

Like I imagine Bill and Jon doing.

Given this thought, I think the #NotAppy-ness is not a go at “punching downwards” as thought above, but more like picking on a friend. You’ll see quite a bit of it if you follow me on Twitter and even more so if you follow Joe Onisick.

There is certainly a line between razzing in a humorous way and public mockery. Where that line stands is another opinion; one only discovered mutually between two people.

Do I want us to continue to outsell NetApp? Of course. But I also rely on the Sales org to worry about that part.

Me, I build communities. That makes this conversation a whole lot easier.

We are all one community.

Today a coworker asked me about extending an EagerZeroedThick disk, as they had a customer inform them that extending an EagerZeroedThick disk actually turned it LazyZeroed. Although this is somewhat true, it is not 100% accurate. What actually happens is the extension becomes LazyZeroed but the original portion of the disk still remains EagerZeroedThick. This is actually by design, if you want to extend the VMDK and ensure that it is all EagerZeroedThick you must specify a flag in the vmkfstools command to do so.

Extending an EagerZeroedThick VMDK

# vmkfstools -X 10G -d eagerzeroedthick /path/to/vmdkfile.vmdk

In the above syntax example it is critical that you specify eagerzeroedthick as the disk format.

jfruh writes "A lot of ink has been spilled explaining to Boomers and Gen Xers how they can best manage, motivate, and retain talented members of the Millenial generation on the job. But it's a two-way street, and those born in the '80s and later could also use a lesson on how to best communicate with older co-workers, who after all will determine their promotion and pay raises for the foreseeable future. Advice includes: make actual phone calls, mirror the level of formality your co-workers use in e-mails, and for Pete's sake don't ask them things like 'R U going?' in a non-texting medium."

Read more of this story at Slashdot.

wabrandsma writes, quoting Apple Insider "The technology, detailed in a patent awarded to Apple on Tuesday by the U.S. Patent and Trademark Office, deals with so-called 'access inputs' that determine what apps, device services, and functions can be accessed by a user. Apple's U.S. Patent No. 8,528,072 for a 'Method, apparatus and system for access mode control of a device,' describes a system that creates user access modes guarded by predetermined gesture inputs." Reading the patent, it appears Apple managed to patent allowing access to some programs without a passcode from the lock screen of a device while protecting others, so e.g. you can quickly swipe to make a phone call or control your music, but have to enter a code to read your email or access your word processor documents.

Read more of this story at Slashdot.

Anita Hunt (lissnup) writes "Hot on the heels of Dave Cameron's demands to make such content universally 'opt-in,' the Independent reports 'Westminster computers were prevented from accessing sex sites 114,844 times last November alone and on 55,552 in April, while February saw just 15 and in June officials blocked 397 attempts.' No explanation has been offered for the variation, although it would be interesting to know if the fall in the number of recorded/reported attempts coincides with the date the FOI request was filed."

Read more of this story at Slashdot.

mattydread23 writes "Data broker Acxiom did something a little unusual this week. It launched a service that lets you see the data they've collected on you. CITEworld writer Ron Miller checked it out, and found it to be mostly laughably inaccurate. Among the things they got wrong included his religion, his interests, and the number of kids he has. But worst? It pegged him as a Windows user."

Read more of this story at Slashdot.

As a profession, we have the perfect scam for our customers. Every time technology marches far enough ahead, and support for the old technology wanes, we convince management to allow us to rewrite the widget, or better yet start over from scratch. Over 30 years. I personally have rewritten the same Mortgage Backed Securities (or equities, or Corporate Bonds, or Treasuries) trading system for a variety of customers in C, C++, Java 1.2, Java 1.4, Java 1.5 and Java 1.6. Oh, they ask for a new feature here and there, but the bulk of the functionality never changes; they just keep paying us to do what is essentially the same work over and over. It's an unintended scam truly worthy of the descriptor: caper!

Capes are cool. They create an aura of mystery. Who doesn't feel it when they see Batman swooping in? Or Superman leaping tall buildings in a single bound? Capes are even more awesome when worn by someone that embodies evil. For example, Count Dracula. Or Doctor Doom. Of course, not all capes inspire awe; some inspire fear, dread and despair...

Derek E. works for a large national bank which has a Lotus Notes workflow application called the Creative Accounting Posting Engine (CAPE), whose purpose is to help guide commercial loan applications through approval and underwriting steps in various departments located around the country. CAPE has been in use for the last 13 years with 600 users and has mostly worked. Some users complained about it being slow, and management has had a difficult time in getting reports out of it.

Since Notes is non-relational, reporting was usually accomplished by designing programs to loop through the data. There were usually ~30,000 documents in the database, so on a good day, CAPE reporting could give a crippled snail stuck in molasses on a cold winter day a run for its money. It was originally written using Notes 4, and was ported to various versions of Notes 5, 6, 7, 8 and now 8.5.

Through the years it was modified countless times by dozens of different programmers and consultants of varied skill levels. As one might expect, the code is a Pit-o-WTF^TM comprising a vast collection of @Formulas and LotusScript programming tucked away in functions saved in various places in forms, sub-forms, events, buttons, and even script libraries. The general programming style is "object oriented if someone was in the mood".

Several years ago, a large project was put in place to re-platform it using Visual basic so it could utilize relational databases. Since LotusScript is a dialect of Visual Basic, the Notes developers should have had a fairly decent handle on it. The Notes developers were trained in Visual Basic and turned loose. Given the Steaming-Pile-o-WTF^TM they produced, perhaps they should have been kept on a tight leash. The project became legendary for the epic level of fail it had attained. The rewrite was never used. As a result, most of the developers and managers quit.

Sadly, Derek inherited the mess.

For the last 2.5 years, Derek worked on several small to medium sized modifications to CAPE. Most of these involved a pop-up input dialog box with scrolling lists of input fields to allow entry of up to 99 different sets of data. The data would then be stored and reported in a variety of sort orders on another form. Since Notes isn't relational in nature, the developers had decided to store the sets of data as tilde-delimited strings: field1~field2~field3~...~field99. Naturally, this was implemented in any number of inefficient algorithms, all just a little different, in every place that this sort of functionality was required. This could be replaced by more up to date techniques like embedded views or XPages, but the amount of code that would have to be located and changed simply made it too risky.

Last year it was announced that CAPE would be re-platformed onto ASP.Net using Agile methodology. A department of existing .Net developers was salivating at the chance to port CAPE to showcase their .Net and Agile knowledge. Unfortunately, it would be their first project developed using Agile methodologies. They seemed to have used Joel Spolsky's "Things You Should Never Do..." articles as a how-to guide because if it was possible to do something wrong, they did.

Since this rewrite was expected to continue for several years, Derek was asked to take on one final Lotus Notes CAPE project; to interface it with a distributed data feed to bring in data for 1-99 customers from a series of three web services: Customer, Account and Exposure, using the same tilde-separated-values data entry/collation/storage mechanism as before. Thus, if someone entered data for 50 customers, each of whom had 12 accounts, it would make 600 sets of web service calls before it could crunch the data and begin to give feedback to the user. The initial estimate was nine weeks. Nine months later it was mostly done, and ran like a recovering crippled snail stuck in molasses on a cold winter day.

After more than a year, the .Net team completed their first set of Agile story-boards and have finally started writing some actual code...

The programmers have pulled off the caper again; after 5 major Notes-ports and a failed full-rewrite, they managed to convince management to pay for yet another full rewrite using yet another set of technologies.