Unhandled expections are not being routed to my custom error page, why?

My understanding is that if you added `app.UseExceptionHandler();` and give it a path that ASP.Net is supposed to load that page when ever there is an error that is not caught in code but in my case I am still getting the normal "Http 500 Internal Server Error" page. I can take the path I give `UseExceptionHandler()` and put it right in my browser and it load the page fine so I know the path and the page work. Am I miss understanding how this is to work, is it broken, or am I doing something wrong?

Startup.cs:

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            //if (env.IsDevelopment())
            //{
            //    app.UseDeveloperExceptionPage();
            //}

            app.UseExceptionHandler("/Error/PageNotFound");
            app.UseIISPlatformHandler();
            app.UseSession();
            app.UseStaticFiles();
            app.UseStatusCodePagesWithReExecute("/Error/PageNotFound");
            app.UseMvc();
        }

Error Page Code:

<p>
    @ViewData["ErrorMessage"]</p>

Please note that "Page not Found" errors are routed to the `/Error/PageNotFound` with no problems, it is just other errors are not. As a test I copied the string from `UseStatusCodePagesWithReExecute` into `UseExceptionHandler` but still get the generic 500 error page.