Writing directly to a FileStreamResult from a crypto stream

I'm working on a web based file sharing project where one of the goals is to avoid ever having the unencrypted data of a file ever hit the disk (including in temp space for spooling). I've managed to get a Middleware in place to handle the chunking of the uploads through the Aes encryptor, however I can't seem to figure out how to do the reverse using the FileStreamResult type for my download controller endpoint. I could technically just decrypt into a memory stream and feed that stream handle in as the FileStreamResult source stream, but I want to avoid loading the entire file into memory on a web server just to buffer it for downloading (we could be dealing 8+ GB files)

Current Code I have:

        [HttpGet]
        public async Task<FileStreamResult> GetAsync(int id)
        {
            var file = await _unitOfwork.FileRepository.GetByIdAsync(id);

            using (var fs = _fileStorageProvider.GetReadStream(id.ToString()))
            using (var decryptor = Aes.Create())
            {
                var pdb = new Rfc2898DeriveBytes(file.EncryptionKey, file.EncryptionSalt.ToByteArray());
                decryptor.Key = pdb.GetBytes(32);
                decryptor.IV = pdb.GetBytes(16);

                using (var cs = new CryptoStream(fs, decryptor.CreateDecryptor(), CryptoStreamMode.Read))
                {
                    //This results in a 500 error
                    return new FileStreamResult(cs, "application/octet-stream");
                }
            }
        }

Error I'm getting in the output window when attempting to download:

Microsoft.AspNetCore.Mvc.Internal.VirtualFileResultExecutor:Information: Executing FileResult, sending file as
Microsoft.AspNetCore.Server.Kestrel:Error: Connection id "0HKVKOSU7G9NK": An unhandled exception was thrown by the application.

System.ObjectDisposedException: Cannot access a closed Stream.
at System.IO.Stream.ValidateCopyToArguments(Stream destination, Int32 bufferSize)
at System.IO.Stream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Internal.FileStreamResultExecutor.<WriteFileAsync>d__3.MoveNext()

Any ideas on this one?