Today’s blog post is about export limits in software programs.
Truncated exports are a common problem in many software programs that handle large amounts of data. In order to avoid long loading times that would ruin the usability of a program, software developers like to truncate large amounts of data. This is a reasonable solution that works well in most use-cases. They don’t anticipate a situation where the user would want to export the complete set of data. Many programs don’t even issue a warning to inform the user about the truncation, they simply limit the number of returned result rows to a predefined number and claim to have successfully finished the export.
The same goes for many APIs that return data upon request like REST interfaces. APIs are usually built to answer very specific queries and return a small number of results, so the default export limits are usually much smaller than in server or desktop applications.
When you rely on an export function of a program that you are not familiar with, make sure to check if it has any restrictions regarding the number of observations it can export. If you’re requesting data from the users of a program that handles data, ask them to make sure that any such restrictions are turned off before they run the export.
Also, count the number of records in your data. When your data has exactly 10.000 or 1.000.000 records, it’s very likely that you have hit an export limit and need to go back and fix that before continuing with your analysis.
Benjamin Greve 