It's not that hard to create a C# wrapper class if you need to hold on to pointers that need to be cleaned up by the C code... you can keep a private property reference and call the appropriate C code as part of your Destructor or IDisposable implementation.

If you're wanting to use a C library, yeah, you need to manage how you use that C library, this is true regardless of the language you are using, it isn't magic.