The article presents an fMRI study of brain organization of working memory during the retention of visual stimuli with different emotional valence. We studied how emotional valence of information influences the working memory processes: whether emotions modulate the activity of a single functional system of the working memory or different functional systems of the brain are activated fully or partially, depending on the emotional valence of information. Twenty-one subjects participated. We used images from IAPS and GAPED databases with different emotional valence (negative, positive, and neutral). Participants memorized the original images and performed a change detection task after the working memory retention period (about 10 seconds). We recorded errors, reaction time, and the BOLD-signal (fMRI). We found significant differences, both in the cognitive task performance and in the brain organization of the information retention period in the working memory, for negative stimuli, compared to neutral. Negative valence of visual information reduces the effectiveness of its retention in the working memory. The fMRI data suggest different topography of activation of cortical and deep brain structures during working memory retention of images with different emotional valence which suggests formation of different functional systems in the brain. Comparison of the fMRI study results with the task performance data shows that negative emotional valence of information combined with level of emotional arousal interferes with formation of optimal functional brain systems for this type of cognitive activity.