Architects have always aimed at developing more adequate software to capture the complexity of biological self-organizational systems in architectural form generation. Agent-based modeling (ABM) emerged as a promising approach, where agents can exhibit emergent self-organizing behavior. However, the nonlinearity and feedback loops typically associated with natural systems are not quite achieved using traditional ABMs. This paper investigates the integration of reinforcement learning (RL) - an artificial intelligence technique - to develop reinforcement learning ABMs to enhance architectural form generation. Decentralized multi-agent reinforcement learning is proposed as an approach where agents learn complex strategies that maximize rewards through interactions with the environment to model emergent structures that are typical characteristics of natural systems. The paper reviews relevant biological self-organization concepts that inform architectural objectives, surveys previous ABM research, and proposes a RL framework that provides a systematic approach for developing RL models that capture the complexity and adaptability of natural systems, focusing on architectural form generation.