Flow graph corpus (FG corpus)

Overview

One of the ultimate goals of the natural language processing (NLP) is to "understand" languages.

However, "understanding" is a difficult problem because it cannot be defined easily. We address this problem by researching procedural text. A procedural text is a set of instructions that describe how to make something, and we can represent this as a graph structure.

Our research focuses on recipes as procedural texts. Millions of recipes have been published on the Internet and in books. It is also easy to find cooking videos.

To represent the "understanding" of cooking recipes we use a flow graph structure. We construct a recipe flow graph corpus (r-FG corpus) that represents the structure of recipes and publish them. The following are possible future applications of the corpus.

More intelligent recipe search compared to keyword matching,
Assistance for cooking by using observed features of human behaviors,
Automatic alignment of cooking videos and recipe flow graphs (cf. KUSK Dataset),
Automatic cooking machines,

and much more.

Definition of Recipe Flow Graph

Generally, publised recipes consist of following components.

Title.
Ingredients.
Steps.

We construct r-FG corpus to represent the structure of "Steps". An r-FG is represented by a rooted directed acyclic graph (a rooted DAG), which is composed of:

Vertices: expressions in the recipe domain (called recipe named entities; r-NEs).
Arcs: relations between two r-NEs.

For example, the root of r-FG in the above figure, which corresponds to completed dish, is "bake/Ac".

Vertices

r-NE tags.
Tag	Meaning	Remarks
F	Food	Include pronoun expression and intermediate/completed food.
T	Tool	Cooking tools, dishes, etc.
D	Duration	Include round numbering.
Q	Quantity	Include round numbering.
Ac	Action by the chef	Separate inflectional endings.
Af	Action by foods	Separate inflectional endings.
Sf	State of foods	Taste, minced, sliced, etc.
St	state of tools	Temperature, size of tools, etc

Each vertex of a flow graph corresponds to a concept represented by a word sequence in the text and a concept type such as food, tool, action, etc. We call them "r-NE".

r-NEs are non-nested and non-overlapping as well as general named entity.

r-NEs are annotated with not only naun phrases (Foods, Tools, etc.) but also verb and adjective phrases (Action by the chef, State of food, etc.). We separate inflectional endings from adjectives, verbs, and auxiliary verbs (Ac, Af).

We annotate r-FG corpus with r-NEs (verticies) and relations of two r-NE (arcs), and we publish the corpus which are annotated with only r-NEs. You can use r-NE corpus as training corpus for our named entity recognizer (PWNER), that can recognize r-NE automatically.

Arcs

Arc labels.
Label	Meaning
Agent	Agent (subject)
Targ	Target (direct object)
Dest	Destination (indirect object)
T-comp	Tool complement
F-comp	Food complement
F-eq	Food equality
F-part-of	Food part-of
F-set	Food set
T-eq	Tool equality
T-part-of	Tool part-of
A-eq	Action equality
V-tm	Head of a clause for timing, etc.
other-mod	Other relationships

An arc between two vertices indicates that they have a certain relationship. In an above figure of flow graph, arcs are indicated by arrows. An arc has a label denoting its relationship type. We denote four types of relationships as follows:

Predicate argument structure (including zero anaphora)
Predicate argument structure is the most important because of the main component of steps of recipes is the relationship between "foods" and "actions" (Agent, Targ, Dest, T-comp, F-comp).
Reference relationship between two foods or two tools
This type associate a food/tool which has emerged the past once or has processed and first appeared food/tool (F-eq, F-part-of, F-set, T-eq, T-part-of).
Relationship between two actions
Sometimes an action verb is repeatedly written just to specify an object such as "carrot you've cut." This verb to the verb which the chef has to really execute as follows (A-eq).
Some clauses, whose head is a verb phrase annotated with Ac or Af, specify the timing or the condition of another action (V-tm).
Other Relationships
All the other modification relationships fall into this category (Other-mod).

We annotated arcs and labels with not only verticies in same sentences but also different sentences. Thus r-FG graphs can represent the whole structure of steps of the recipe. For example, if you trace arcs of a food , you can trace the cooking process of the food.

Notice: Arc label names [subj, d-obj, i-obj] in past papers are changed to [Agent, Targ, Dest] respectively.

Download

r-FG corpus Ver.2.0 (under construction)

This is a toolset for r-FG corpus. Note that this toolset does not include original recipe sentences but only annotation information. You can get original recipe sentences from "Cookpad Data Set". You can make r-FG corpus and r-NE corpus by using Both of them. Please see 00readme.txt in the toolset for details.

If this toolset does'nt work, please contact the author.

r-FG corpus

Steps of recipe are represented by a flow graph (r-FG/r-NE corpus does not include title and ingredients). Real files are shown in the right figure (csv format). One recipe corresponds to one file, one line corresponds to one vertex.

The followings are elements of a recipe flow graph file.

A: ID
B: Character-position in the sentence
C: r-NE tag
D: r-NE word sequence
E-Q: Arc labels and indicating vertex information

r-NE corpus

We annotated following information to the preparation of recipe text. Real files are shown in the right figure (tree format, available for PNAT). One recipe corresponds to one file, one line corresponds to one word. The followings are elements of a sentence in r-NE corpus file.

Word ID
Head ID (dummy)
Word
Part-of-speech (automatically)
Pronounce (automatically)
r-NE tag (IOB2 format)

Corpus Specification

Corpus specification. (Ver 2.0)
	Source	#Recipe	#Sentences	#r-NEs	#Words	#Characters
r-NE corpus	Random	400	3,058	17,635	55,852	84,536
	Nikujaga	36	259	1,604	4,690	7,024
	Chikuzenni	1	11	76	240	341
r-FG corpus	Random	208	867	8,316	23,357	140,473

Latest version (Ver 2.0) is shown in the right table. r-FG corpus is a subset of r-NE corpus. We are enlarging the size. We aim to construct r-NE corpus consists of 1000 recipes and r-FG corpus consists of 500 recipes.

KUSK Dataset 2014RC

Kyoto University Smart Kitchen dataset (KUSK dataset) is a dataset of activity observation in a kitchen. KUSK dataset have 20 real-cooking videos from 20 recipe. 20 recipes , which are the data source of KUSK dataset 2014RC, are included in our corpus.

Members

Yoko Yamakata
Shinsuke Mori
Tetsuro Sasada
Hirokuni Maeta
Takuya Kadowaki

Past members

Takashi Yamasaki

Reference

A Framework for Procedural Text Understanding,
Hirokuni Maeta, Tetsuro Sasada, Shinsuke Mori
IWPT, 2015.

KUSK Dataset: Toward a Direct Understanding of Recipe Text and Human Cooking Activity Information
Atsushi Hashimoto, Tetsuro Sasada, Yoko Yamakata, Shinsuke Mori, Michihiko Minoh,
CEA, 2014.

Flow Graph Corpus from Recipe Texts
Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, Tetsuro Sasada
to appear in LREC, 2014.

Structural Analysis of Cooking Preparation Steps in Japanese.
Reiko HAMADA, Ichiro IDE, Shuichi SAKAI, Hidehiko TANAKA.
The fifth International Workshop on Information Retrieval with Asian Languages, pp. 157–164. 2000.

Control Structures for Actions in Procedural Texts and PT-Chart
Yoshio Momouchi.
The 8th International Conference on Computational Linguistics, Vol.1, pp. 108–114. 1980.

Acknowledgements

In this research, we used recipe data provided by Cookpad and the National Institute of Informatics.

Change log

2015/07/01: Arc labels [subj, d-obj, i-obj] are changed to [Agent、Targ、Dest].

Last Change: 05/12/2016 by Tetsuro Sasada

Kyoto University ACCMS NLP Group