You may find discussions by grammarians on semantic sub-types of Karman useful in this respect.  Works from Bhartr̥hari's Vākyapadīya to Kauṇḍabhaṭṭa's Vaiyākaraṇa-Bhūṣaṇa and Nāgeśa's Laghumañjūṣā discuss these sub-types.  The discussions on verb semantics classifying verbs into kartr̥stha-kriyaka versus karmastha-kriyaka and kartr̥stha-bhāvaka versus karmastha-bhāvaka may also provide some clues.  I have discussed the effects of some semantic sub-types of Karman [affected versus non-affected] in the syntax of passives of dvikarmaka verbs.  You can see this in a sentence like ajāṃ grāmaṃ nayati > ajā grāmaṃ nīyate; but not ajāṃ grāmaḥ nīyate.  On the other hand, grāmaṃ gacchati can be passivized as grāmaḥ gamyate.  So grāma in relation to nayati and gacchati seems to have a somewhat different perception.  With respect to nayati, it may be more or less adverbial as you say.  On the other hand, with respect to gacchati, it has some closer semantic connection.  This looks like a great topic for deeper introspection.  Perhaps Hans Hock and George Cardona could add to this discussion.  With best wishes,

Madhav