I think this might be an issue similar to the one I had a number of months ago - apologies if it's not a fix or I misread your current problem.
In your extension, make sure you point your reference at the Newtonsoft version inside the Outsystems instalation folder, eg C:\Program Files\OutSystems\Development Environment 11.0\Integration Studio (references -> add reference -> browse).
Outsystems comes packaged with this verson of Newtonsoft, and I can only assume there's something hardcoded in the depths of it that expects this specific file / version. I don't really understand what's going on, but when we had our issue it sorted it for us. Of course, being Integration Studio, this isn't documented anywhere, so we had a dozen extensions using Newtonsoft from nuget for months without issue... until we did a platform upgrade and everything broke.
I don't believe we are talking about the same problem. The issue was that I was using a third party library that uses a later version of Newtsonsoft.Json. Outsystems doesn't allow using versions that are newer than the hardcoded version it uses. Speaking with support back in August I confirmed with them that there is no fix for this except using the version they use. I couldn't do that because I didn't have the option of modifying the third party library I was using and it calls functions that don't exist in the Newtsonsoft version that Outsystems uses.
My own solution to this was to just give up on the extension and find another way to do what I needed from within a regular web module.