We just realized that by adding SmartObjects to the SmartObject OData API we'll bypass any category security settings in K2 configured for this SmartObject or in other words: anyone is able to read any data from the added SmartObjects. In our example personal documents will be stored there, so this is an absolute no-go.
The current workaround is to create a separate service instance for the same database, but configured in impersonation mode to pass through the user credentials to the backend system and configure the permissions there. This might work in a very limited scenario with only a few users and with a SQL based SmartObject, but will certainly not work in all backend systems like SAP because it's not very common to use Active Direcory as the authentication provider for SAP.
The ODATA API must respect the category security settings, otherwise in my eyes the API is not fully implemented and a major security risk!
The problem described is not specific to the ODATA API - SmartObjects do not currently have an EXECUTE permission and it is essential to implement security within the backend system using the authenticated user's context or passing in the K2 user context as described in the "Do not pass user identity from the user interface" section of this article: https://community.nintex.com/t5/Best-Practices/Securing-K2-Solutions/ta-p/125864. Leaving this idea open for voting, but implementation would be around security for EXECUTE permissions on SmartObjects, which the Authorization Framework doesn't include at this time.