As per Pāṇini's rules, a word-final m changes to anusvāra before any consonant [mo 'nusvāraḥ, hali].  Then come option rules.  Before y/v/l, an anusvāra can optionally change to nasal versions of y/v/l.  Before the stops, it can optionally change to a nasal homogeneous with the following stop [parasavarṇa].  This leaves r/ś/ṣ/s/h, and before these it remains anusvāra.  In some Vedic traditions, in this last environment, it becomes nāsikya or raṅga.  Then there are rules which change n into an anusvāra in some environments.  This is the general description according to Pāṇini.  Some other grammars do allow an m occurring at the end of an occurrence before a pause to change into anusvāra, reflecting some local variation.  Of course, the actual pronunciation of an anusvāra probably differed regionally as we notice today, and different Vedic traditions have conventionally settled ways of its pronunciation.  Also rules that are optional in Pāṇini don't necessarily remain optional in various Vedic tradition.  In most Vedic recitations I have heard, the change of an anusvāra into a parasavarṇa nasal is almost done invariably.

Madhav