State Space Construction using Image Input
To be applied in real world, action learning theory, such as reinforcement learning, needs pre-processed sensor data that is adequate for practical learning. We propose a method of sensor data categorization to enable the construction of state space directly from high dimensional input sensor vectors. In this research, we assume that an evaluation is given to each step of action; whether the action was good or bad. We propose to categorize input space and learn adequate action from binary evaluation. The basic idea of state space categorization is to distinguish one state from another state, when the evaluations of the same action taken in two states are different from each other. When the evaluation changes to 'bad', the agent makes discrimination between these two states, before and after the change of the evaluation. Input vectors are expressed as weight vectors of a 'node' based on vector quantization theory. Here we call these nodes “state nodes”. We use topology representing networks (TRN) algorithm, in order to make use of topology in the input space. The idea to modify the action under the binary evaluation is that an action can be replaced by another if an action is found 'good' in common with other states. We realized the idea of action modification by using Radial basis function (RBF). The proposed method is applied to a pushing task of a round object by a manipulator. Vision information is used as inputs from the sensors. Experiments utilize the result of a simulation. In the simulation, constant number of nodes were created from high dimensional inputs. In the experiment, adequate pushing action that absorbs the difference of the motion of an object between the simulation and the real world is achieved.
Keywords: State Space Segmentation, Radial Basis Function, Image Input
Fig.3 Experimental environment Fig.4 Result of simulation