Is there a way to avoid initializing database tables on each Orion startup?

Yes, there is. Add this to your settings:

prefect config set PREFECT_ORION_DATABASE_MIGRATE_ON_START='False'