When it comes to AI Model training, more data is usually better. However there are times when simply adding more data isn't the best way to maintain or increase your model's accuracy in making predictions.
Data volume, data quality, label, quality, and data balancing are all important when trying to get the best possible training score. Read more about this here.
Your training score is calculated based on a subset of the data that is excluded in training. Put another way, you might upload 100 pieces of data to be trained, but some of those 100 pieces will be automatically set aside for the model to test itself on during the process. This is what delivers the performance score.
When you add more data, this also adds more data into this testing subset, which can lead to a different performance score. Not because the AI Block is decreasing in performance, but because the performance score becomes more accurate (assuming you've added good quality data, of course).
This testing subset is also shuffled each time - it's not always going to be the same data that is used to generate the performance score, so that can sometimes change this value too.
We are always working on new ways to add clarity to this process so that you can continue to improve upon your results.
Create your own standardized metric for measuring performance
The Performance Score is provided to give you a good understanding of how well your model might perform in a real situation, but there is no substitute for the real thing. The only robust truth can come from testing it yourself on what you want to test.
We recommend building your own small dataset for testing that you can use to check the performance. These might be a random set of images, text, or PDFs that doesn't change, that you keep in a local folder on your computer, and that does not include in the uploaded data going into your AI Block.
You can then use these data items on the Test part of the process once you've retrained, and use that to gauge your own improvements in prediction quality.
Need more ideas on how to measure and improve the effectiveness of your AI Block?
We’re available to provide advice, answer questions, and talk through setup options whenever you’re ready. Click here to read more about getting in touch.
