TcpClient connection slower than .NET?

Before I start wasting time with code a quick summary.
I wrote a HttpClient for some remote service we have to connect to.
Wrote it all in a NET Standard 2.0 library and then wrote two console apps which utilize the library to create an instance of the client and then connect to that server using TLS...client cert and eventually send and receive data.
To my surprise after they both create an instance the connect call in a NET Core app is always 100ms slower. 
NET console -> 200ms, Core 300ms. Every damn time. Same code.
Now this isn't anything terrible or something that's probably gonna kill us(we are expecting thousands of call though) but I was quite surprised.
Does anyone know if there's a setting or something that could at least bring the speed to .NET average times? Perhaps there are some defaults that are different in TcpClient itself in NET Core which specifically prevent it from connecting as fast?
I really was expecting .NET Core to be faster or at least just as fast here since it doesn't have the burden of supporting old code.