I don't really know if this is doable, but currently the Expo SDK always ships with GoogleMaps / Segment / Branch ... and maybe other libs later (like Intercom?)
Not increasing too much the SDK size by including every lib out there is a concern, yet the more libs are included the more usecases would be covered.
I think it would be nice if some SDK libs were opt-in. The Expo client's purpose is mostly to provide a good DX so it could embed all plugins, while the standalone apps could allow to remove some unused native code.
The plugins would only be "official ones" to reference in expo config, and not done by random users, otherwise the Expo client wouldn't have the code to run the all the apps. Allowing custom plugins, leads for me to a quite messy experience (based on experience with Phonegap Developer App, which is the Cordova equivalent of Expo client): people needing custom plugins could simply detach.