I'm subject to notification requirements for public exposure offences. These are the basic notification requirements, I don't have any special conditions above these. My sister's ex boyfriend, the biological father of my 8 year old nephew and step father to my 3 year old niece called police with concerns for his son's safety due to my sex offending history and that I was staying overnight at times at my sister's home. I've got less than 3 years left of my 7 years notification requirements.
The police visited as a result of this reported concern, my mum, sister and myself were furious at this childish behaviour and the police left with zero concerns. Social services were then brought in and they too, after one visit to my sister and kids left with zero concerns. I can have all the contact I want and stay over with or without supervision. My offences weren't against children, also there was no allegation of anything made by my sister's ex, only that he had serious concerns.
If I can have contact with children with or without supervision, then I'm miffed about your situation with your partner since no charge was made and any risk perceived has to be explained by social services as to what exactly the risk to your children from your partner is likely to be. As stated in previous replies, every case is different and I wish you all the best in finding a satisfactory outcome where you, your partner and children can live a good life with your ex partner's concerns not an issue.
Also, don't sign any agreement without first consulting a solicitor with a reputation of dealing with these kinds of cases. They will try to get away with anything if you let them. Terminology and how they say things in such agreements can be vague and/or ambiguous and a solicitor will annihilate these. You need a solicitor to write to the social workers concerned asking why they feel there is a risk then address any agreement put forward.