IIS certificate private key

I'm trying to load the private key from a certificate stored in the userstore.
The user under which the Application Pool is running is not an administrator.
It seems the user's profile is not loaded(even though that option is set in the pool) so I get this error:

Application startup exception: Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Key not valid for use in specified state
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__61_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)

When running as an administrator this problem doesn't exist and from all that I've managed to get online is:
-when cert is in the user store, you need the profile loaded
-when cert is in the machine store you need to be running as an admin or have access to the store.
Which didn't help at all because when running as administrator that certificate is also stored in the user store and it works just fine.
Does anyone have a solution for this?
Somehow loading the user profile or at least a different way to get the private key under a non administrator account?
Perhaps there are more permissions that need to be set to get the private key...but as far as i remember you get a completely different exception when it's a permission issue.
Any help would be most appreciated.