I have a system that generates alot of logs. (just about 500.000 lines of logs per hour (only about 5.000 of those lines will be saved and used)
I want to be able to make some statistics with these data, to find some specific issues.
As this will be run from a Linux machine, I am thinking that MVC core would be the way to go.
Well, the question is, how should I insert the log data to the database?
1. Have either a perl script, or ASP core running on this or another machine, filter the needed data and insert it through the MVC core app, with a [httppost] in the controller, and post the data in a XML or JSON format, and let the MVC core app take care
of inserting it to the database.
(One advantage I see here, is that if I decide to change the database model, I will only need to change it in the MVC Core app which eases the maintenance)
2. Have a perl or ASP core script insert the data directly into the database
3. Have the MVC core app read the log files in a background job scheduler (if at all possible)
4. something else, that I haven't considered!
What is the most correct/best way to achieve this?
I am thinking 1, but is there something that I have completely misunderstood?
Thank you for your help
Regards
Sean