Autonomous driving carries the promise to drastically reduce the number of car accidents; however, recently reported fatal crashes involving self-driving cars show this important goal is not yet achieved, and call for better testing of the software controlling self-driving cars. To better test self-driving car software, we propose to specifically test critical scenarios. Since these are difficult to test in field operation, we create simulations of critical situations. These simulations are automatically derived from natural language police reports of actual car crashes, which are available in historical datasets. Our initial evaluation shows that we can generate accurate simulations in a matter of minutes.