We are in the process off moving our 2008R2 VM's from the 2008R2 HyperV servers to new Server 2012R2 Hosts.

We shut down the VM's copy the files and VHD's to the new CSV's en import the VM in the Hyperv Manager. Then we make them high available in the Failover Cluster Manager (Configure role - Virtual machine). We mount the integration tools and update the VM to version 6.3.9600.16384

For a specific type of VM (mostly RDS Host servers) we always had Dynamic Memory configured (when they were hosted on de 2008R2 platform), so we are using the same settings on the 2012r2 platform. The memory settings were;

Startup memory: 1024 MB

Minimum memory: 1024 MB

Maximum memory: 12288 MB

These VM's reboot every morning, this is done for specific reasons. But now once in a while (once per week/2 weeks) we notice that the VM's are not using more memory then 1024 MB while the demand is much higher. Rebooting the server helps most of the times, live migrating to another host also helps. In the VM we see that memory usage in the taskmanager is 99-100%, and after the move it immediately starts using more than the minimum configured amount.

Until the failover the memory usage was 1024 MB and it did not get any higher.

This happened several times. Last week we changed the Memory configuration to:

Startup memory : 2048 MB

Minimum memory: 2048 MB

Maximum memory: 12288 MB

But this morning we had a call about the performance of one of the VM's, We saw that it was only using 2 GB memory while the demand was much higher. After live migrating it to another host it started using more memory immediately.

The 2012R2 hosts are not overcommited, there is a lot of memory still available for the VM's. Those VM's never had this problem on the 2008R2 Hyperv platform.

Any idea why this happens?

Peter Camps

• Edited by Peter Camps Monday, November 24, 2014 12:13 PM

As you have shifted these machines over to Server 2012, have you been updating the Integration Services? It almost seems to me the server never comes out of the start up state... Also for another test lets try and do start-up memory of 2048 and minimum memory of 1024. I'd be interested to see if the memory falls back to 1024 after the initial start-up.

In the Hyper-V Application logs on the host in the windows event viewer do you see any errors with dynamic memory?

In the guest go to perfmon and view these performance counters if available.

• Hyper-V Dynamic Memory > Guest Visible Memory
• Hyper-V Dynamic Memory > Physical Memory

Also, in the hyper-v console does it show memory status as OK and what is the assigned memory vs memory demand.

• Edited by Jugganutz187 Monday, November 24, 2014 10:27 PM

Hi,

Yes, all the VM's were upgraded with the latest Integration Tools (6.3.9600.16384).

On the hosts i don't see any error in the eventlog concerning Dynamic Memory, in fact the latest VM that had this issue rebooted at 04.00 yesterday morning and in the eventlog Hyper-V of the host i see the following Informational message;

The 'Microsoft Dynamic Memory Controller' device in 'VM' virtual machine is loaded and the protocol version is negotiated to the most recent version. (Virtual machine ID 330DF84E-118D-44A3-B57F-F27234E1A513)

All the other events are informational and no error shows in the eventlog.

The performance counters you mentioned are only available on the host, not in the guest. Before i can give you any information about the memory status and these counters i have to wait until this problem occurs again.

I have changed the memory settings of a few VM's to the values you asked.

Now all we have to do is wait.......

Grtx

Peter

Doh, you are right that is a host side monitor. I forgot about that.

Yeah, I think with those settings and some of the information in hand you should be able to catch the issue happening and possibly a log of it.

Are you allowing the virtual machines to span NUMA nodes?

Yes,

NUMA node spanning is enabled.

No problem today (so far)

Hi Peter,

I'm sorry to say but provided screenshot doesn't match with your description.

Based on my test , it seems that the VM's memory drop down after you LM it to another node .It is more like an out-of-memory issue .

I would suggest you to check the memory usage in that VM .

Best Regards

Elton Ji

Hi Elton Ji,

The screenshot shows the following, the customer called us because everything was very slow and work was almost impossible. One of our servicedesk people managed to logon and saw in the taskmanager that there was only 1 GB of RAM available and (off course) completely used.

The high 100% line in the taskmanager was at the time the memory usage was 1 GB. Demand was much higher and no more RAM was given.

At that time i decided to move the VM to another host, and as if something was triggered the VM was assigned more RAM. The line dropped but also the scale off the taskmanager changed. The usage was still 1 GB (and climbing), but the scale now changed to 0-10 GB.

So the usage off RAM in the VM did not change but the scale of the histogram changed. First is was 0-1 GB (0=0%, 1GB=100%) to 0-10 GB (0=0%, 10 GB=100%).

I hope you understand my explanation.

Grtx

Peter

Hi Peter,

>>We mount the integration tools and update the VM to version 6.3.9600.16384

As far as I know , the newest Integration services is newer than 6.3.9600.16384 , please try to update the hyper-v host then update the integration services to check the result .

Best Regards

Elton Ji

Hi Elton,

I recently (last Monday) updated a 4-node 2012R2 cluster with all applicable updates. The integration services version i mentioned earlier is still the current one on that cluster. So i have not been able to find a newer version off these integration services.

At the moment i am updating another 4-node 2012R cluster (where some of the problem VM's are placed), to see if this can fix the problem (including big update KB3000850). Since i posted this problem here it has not occured anymore, so have to wait and see what is happening.

Thanks so far

Grtx Peter

Hi,

It has taken quite a while, but this morning we had one RDS server that was only assigned 2 GB RAM (the startup memory amount), while it was configured to a maximum of 10 GB RAM it did not use more than 2.

At that moment it said in my hyperv manager the following;

Memory Status and Demand were blank. The memory usage in the VM was 100% for more than one hour, i started Perfmon on that Host and viewed the performance counters you mentioned before.

As you can see, all values are 2 GB, the amount that was assigned to the VM. There were no events in the eventviewer on the Host and in the VM itself. I live-migrated the VM to another Host in that cluster and immediately the values changed as you can see down here (the screenshots were made 30 minutes after the live migration).

This cluster is completely up to date with all Windows Updates en HP SPP (firmware and drivers). I am unable to reproduces the issue, but it still happens once in a while.

Any further ideas are appreciated.

Grtx

Peter Camps

• Edited by Peter Camps Monday, January 05, 2015 11:19 AM

Peter,

I think this is a bug of some sort. I say that because the components that make up dynamic memory are as follows.

Memory Balancer(Host service, coordinates how memory changes are made.) This is also what shows the memory demand counter i believe.

Dynamic Memory Virtualization Service Provider (this is included your VMWP.exe proccess, one per VM. Essentially how it runs on the host. He listens to the Service Client for metrics)

Dynamic Memory Virtualization Service Client (this is inside the VM and reports to the Dynamic Memory Virtualizaton Service Provider.)

Since you live migrated the machine it made dynamic memory work on the other host. This means the Service Client is running in the client and shouldn't be an issue. The Memory Balancer is the server and shouldn't be the issue, so this means the "Dynamic Memory Virtualization Service Provider" is in question. When you live migrate the machine its going to create a new VMWP.exe process on the clustered server. So now the question is it the host that couldn't listen to the service or the worker process skipped a beat and has a bug.

Out of curiosity does it happen to both hosts? Also have you profiled the servers to see how much memory they really require on start-up? When you reboot the RDS servers, how many VM's do you reboot and is it a staggered process?

Hello,

We have seen it happen to several different VM's (2008R2 OS in those VM's) and on several different Hosts. Even on different Clusters (all Server 2012R2 Hosts).

Because of this issue we have updated all our clusters, but did does not seem to have resolved it. Some of the VM's we migrated from our old 2008R2 clusters and some were newly installed on the 2012R2 cluster.

Most of the hosts have 2 to 3 VM's that reboot in the morning, Some reboot at 04:00 AM, others a bit later. The hosts normally have 25 to 50 GB RAM available when all the VM's are running, so there is plenty off avaliable RAM.

We haven't profiled the servers to see what the server requires at startup. Mainly because we have never seen this issue on our 2008R2 cluster (with the same settings), and it only happens occasionally.

Hello,

We have seen it happen to several different VM's (2008R2 OS in those VM's) and on several different Hosts. Even on different Clusters (all Server 2012R2 Hosts).

Edit (19-05-2015) **** We have also seen the problem happen in 2012R2 VM's ****

Because of this issue we have updated all our clusters, but did does not seem to have resolved it. Some of the VM's we migrated from our old 2008R2 clusters and some were newly installed on the 2012R2 cluster.

Most of the hosts have 2 to 3 VM's that reboot in the morning, Some reboot at 04:00 AM, others a bit later. The hosts normally have 25 to 50 GB RAM available when all the VM's are running, so there is plenty off avaliable RAM.

We haven't profiled the servers to see what the server requires at startup. Mainly because we have never seen this issue on our 2008R2 cluster (with the same settings), and it only happens occasionally.

• Edited by Peter Camps Tuesday, May 19, 2015 7:51 AM

I'm curious if there is a lag or timing thing because of ram constraints and the amount of services on start-up... To test a theory could you build a VM and maybe script some of these processes with a simple batch file to launch at system startup? I'm thinking for memory testing start with the start-up amount or slightly greater of the startup memory value. See if you can get a consistent result of it breaking while you reboot. If yes i think it would be a bug.

Sorry for the shot in the dark. If I recall when the issue is happening all the hyper-v services are started correct?

Hi,

I have the exact same issue with my setup. Is this a confirmed Bug or are there any solution? 

Hello,

We have opened a case with Microsoft concerning this issue. We have also discovered that it also happens with 2012R2 VM's, at first we thought it only happened with 2008R2 VM's but this is not true.

At this moment we are collecting information, performance logging, we have done several MSDT inventory on VM's and hosts. So far there is not yet a solution but i hope to have some positive news the next few weeks.

As soon as i have a solution or important facts about this issue i will post it here. But at this moment i think this will take a while.

If somebody as has any news or information, please share it with us. It has been a problem for quite a while now, and we would like to have this resolved.

Grtx

Peter Camps

• Edited by Peter Camps Tuesday, May 19, 2015 7:48 AM

This is looking to be a bug.

The MS product group are currently investigating... It is a timing issue around the Dynamic Memory driver on boot of the VM. We have sent them a couple of crash dumps, some ETL tracing and a copy of one of our VMs!

They are currently testing multiple reboots in the lab environments. Luckily a couple more people have raised cases recently about it so its getting some traction.

We have raised as SEV B so are in conversation daily regarding the issue. As soon as we have a fix i'll post more info.

@ Peter Camps - Could be a good idea for us to share case ID's? sounds like we are quite a few steps ahead of you in the PSS case?

Hello MS Jim,

Great to hear from you, our case ID is REG:115050712710411. Thus far i have not ben able to reproduce it myself by rebooting our testserver we created. So have to wait until it happens again on one off the production servers.

@ Microsoft Mike - If you could give me your ID, than i can inform our support contact at Microsoft and let him know about your case.

Grtx

Peter Camps

Hello,

Yesterday we heard from our Microsoft support contact that this case (and many more) have been elevated to a higher level. It is recognized as a bug and a specialized team is working on reproducing the problem and finding a definite solution.

Hope to hear soon from them.

Yeah, I think its my case which is at the top of the tree at the moment. Actually on the phone to one of the Senior Escalations Engineers currently (and hes speaking to the product group).

My case is  [REG:115042212661754] Intermittently after a reboot the VMs on the hyper-v host fail to come up with dynamic memory configured though we see that setting in place in the VM settings. Been open since April.

Your case has been discussed with my engineer (Kartikay). FYI he is a *VERY* good engineer.

He was actually already aware of the case as your engineer approached mine for troubleshooting steps :-) We have just tired a few more steps direct from the PG but we already knew the answers. We both suspect the next stage will be a live debug using internal symbols from MS. But there is a LOT of paperwork, NDAs and legals to jump through before we can get there. 

Interestingly they have a copy of one of my VMs but couldn't repro it in their environment (100+ reboots).

Hi,

Yesterday we got a call from Microsoft that a fix was available;

https://support.microsoft.com/en-us/kb/3050265

It is a (optional) client fix (in the VM) and at the moment it is only available for Windows 7 and Server 2008R2, the fix for Server 2012R2 will be released a bit later. At the moment we are installing it in our Server 2008R2 VM's and wait to see if it fixes the problems.

Hi,

Yesterday we got a call from Microsoft that a fix was available;

https://support.microsoft.com/en-us/kb/3050265

It is a (optional) client fix (in the VM) and at the moment it is only available for Windows 7 and Server 2008R2, the fix for Server 2012R2 will be released a bit later.

• Edited by Peter Camps Thursday, June 04, 2015 9:40 AM

Not sure how they came to that conclusion... That "fix" is a replacement Windows Update Client!

I'd tell your engineer to get a grip!

I know, it seems a bit absurd that this is supposed to be the long awaited fix.

I haven't got a reply from them concerning my mail if this is really what we are hoping for. Haven't had a response yet.

Has it fixed your 2k8 issues though?

We have installed the update on a few 2008r2 servers. Hard to tell if it is a solution, because we cannot reproduce it and just have to wait for it to happen again (or not)

Have you heard anything from Microsoft yet?

Hi,

Yesterday we got a call from Microsoft that a fix was available;

https://support.microsoft.com/en-us/kb/3050265

It is a (optional) client fix (in the VM) and at the moment it is only available for Windows 7 and Server 2008R2, the fix for Server 2012R2 will be released a bit later.

This is a joke, right?  I can imagine the look I'm going to get when I go tell the other people on our team that the dynamic memory fix is an update to the Windows Update client.

Just in case someone can use it here is the PowerShell I use to get the VMs that seem to be having the dynamic memory issue.

Get-VM | where {$_.DynamicMemoryEnabled -and $_.MemoryDemand -eq 0 -and $_.ReplicationMode -ne "Replica"}

You can run it on a single host as-is or use the -ComputerName parameter to list multiple/remote hosts.

Got the exact same problem in our environment. Windows 2012 Hyper-V hosts with 800 VMs. I was going to log a case with MS on this issue but will hold off and follow this thread! Hopefully you will find the solution and share it with the rest of us. Good Luck!

Got a call today that the "fix" for Server 2012R2 is also available;

https://support.microsoft.com/en-us/kb/3050267

They also told me that the team is still investigating the problem to see if this is the final solution. I got the idea that this is seen as a workaround until the final solution ia available. Reproducing the issue seems to be the biggest problem. They are going to keep me informed, but encouraged us to install this update in the mean time.

I will let you know if there is any news.

Seriously? That KB article doesnt even list the dynamic memory problem in the issues fixed section and yet they want us to install the patch? Cant take this patch seriously to be honest.

Our case is still with the Product Group in the states...
Usual PG crap, asking for irrelevant logs and information.
Still no further forward but the PG is very busy with Server 2016 - so getting any focus on 2012 is a joke at the moment.

The Windows update client "fix" is a joke, speak to your TAM and get the engineer pulled off your case.

As expected the "fix" has not resolved the problem, yesterday we again had a dynamic memory issue with a server (2008R2 VM) that had this fix (KB3050265) installed.

Informed the engineer who has been assigned to our case and got the following response;

I know that the update is not working fine for solve your issue in the dynamic memory, I just finished a meeting with the developer team and they are testing a possible workaround for fix the issue, they tell me more or less two days more for make sure this workaround is working fine and dont affecting the customers environment.

Have no idea why they let us install this fix in the first place, but hope they are actually working on a definite fix.

Our TAM got back to me last night...
The PG have been given the exact specification of our environment and exactly how to repro the issue. they are knocking up a lab to try and repro in-house. If they cannot repro in-house then we will be performing live debugging on our environment (as I suspected).

I have been through these kind of things before, its going to be time consuming, I suspect I wont hear anything for the next 5-6 days whilst they build the lab and attempt to repro internally.

All the tracing they have had us doing has proved useless as Dynamic memory has practically no diagnostic logging built in by default - doesn't help diagnosing issues!

the fact that this has gone to the PG says that it will get fixed, but don't expect a quick answer! I'm putting my money on the bug being fixed around November / December! They will NOT be finding a fix in the next few days, he\she is unnecessarily raising your expectations only for them to be broken!  

your engineer sounds rubbish. If you have a TAM, ask your TAM to get the case re-assigned. Out of interest what is the PSE's name?

Thanks to Microsoft Jim I've been led here - looks like I've also got issues with Hyper-V and virtual memory not functioning.  I posted a similar question, not finding this thread initially.

https://social.technet.microsoft.com/Forums/windowsserver/en-US/cc4b4672-473b-4ebf-994f-93034dfdff42/some-windows-guests-not-demanding-more-memory

In my case, a live migration of the affected VM will resolve the issue - but not a restart of it.  I'm going to follow this thread and see what comes of it.

Bite the bullet and raise the case :-)

I am 110% certain this will come back as a bug so it will be free of charge :-) More people who open cases the more pressure goes on the PG to fix the issue quickly.

Welcome to the Dynamic memory thread Lester!

Still not much further forward, PG and my escalations engineer have both built repro environments but as of yet haven't managed to reproduce the issue. I suspect they don't have the resources to run at the scale we are? sounds like a couple more people have raised cases for this also, which is waking PG up to the issue. I'll post an update when I know more.

We haven't heard from our engineer or Microsoft in any way. Sent a mail last week and found that the case enigineer is on holiday. Nobody else is replying our mail.

This morning again a mail to MS to build up some pressure. But if i read all the comments here i am not very hopefull that the issue will be resolved very soon.

I'll keep you updated.