Kyoto University ACCMS NLP Group

Flow graph corpus (FG corpus) English

Overview

One of the ultimate goals of the natural language processing (NLP) is to "understand" languages.

However, "understanding" is a difficult problem because it cannot be defined easily. We address this problem by researching procedural text. A procedural text is a set of instructions that describe how to make something, and we can represent this as a graph structure.

Our research focuses on recipes as procedural texts. Millions of recipes have been published on the Internet and in books. It is also easy to find cooking videos.

To represent the "understanding" of cooking recipes we use a flow graph structure. We construct a recipe flow graph corpus (r-FG corpus) that represents the structure of recipes and publish them. The following are possible future applications of the corpus.

and much more.



Definition of Recipe Flow Graph

Generally, publised recipes consist of following components. We construct r-FG corpus to represent the structure of "Steps". An r-FG is represented by a rooted directed acyclic graph (a rooted DAG), which is composed of: For example, the root of r-FG in the above figure, which corresponds to completed dish, is "bake/Ac".

Vertices

r-NE tags.
TagMeaning Remarks
F Food Include pronoun expression and intermediate/completed food.
T Tool Cooking tools, dishes, etc.
D Duration Include round numbering.
Q Quantity Include round numbering.
Ac Action by the chefSeparate inflectional endings.
Af Action by foods Separate inflectional endings.
Sf State of foods Taste, minced, sliced, etc.
St state of tools Temperature, size of tools, etc
Each vertex of a flow graph corresponds to a concept represented by a word sequence in the text and a concept type such as food, tool, action, etc. We call them "r-NE".

r-NEs are non-nested and non-overlapping as well as general named entity.

r-NEs are annotated with not only naun phrases (Foods, Tools, etc.) but also verb and adjective phrases (Action by the chef, State of food, etc.). We separate inflectional endings from adjectives, verbs, and auxiliary verbs (Ac, Af).

We annotate r-FG corpus with r-NEs (verticies) and relations of two r-NE (arcs), and we publish the corpus which are annotated with only r-NEs. You can use r-NE corpus as training corpus for our named entity recognizer (PWNER), that can recognize r-NE automatically.

Arcs

Arc labels.
LabelMeaning
AgentAgent (subject)
TargTarget (direct object)
DestDestination (indirect object)
T-compTool complement
F-compFood complement
F-eqFood equality
F-part-ofFood part-of
F-setFood set
T-eqTool equality
T-part-ofTool part-of
A-eqAction equality
V-tmHead of a clause for timing, etc.
other-modOther relationships
An arc between two vertices indicates that they have a certain relationship. In an above figure of flow graph, arcs are indicated by arrows. An arc has a label denoting its relationship type. We denote four types of relationships as follows:
  1. Predicate argument structure (including zero anaphora)

    Predicate argument structure is the most important because of the main component of steps of recipes is the relationship between "foods" and "actions" (Agent, Targ, Dest, T-comp, F-comp).

  2. Reference relationship between two foods or two tools

    This type associate a food/tool which has emerged the past once or has processed and first appeared food/tool (F-eq, F-part-of, F-set, T-eq, T-part-of).

  3. Relationship between two actions

    Sometimes an action verb is repeatedly written just to specify an object such as "carrot you've cut." This verb to the verb which the chef has to really execute as follows (A-eq).

    Some clauses, whose head is a verb phrase annotated with Ac or Af, specify the timing or the condition of another action (V-tm).

  4. Other Relationships

    All the other modification relationships fall into this category (Other-mod).

We annotated arcs and labels with not only verticies in same sentences but also different sentences. Thus r-FG graphs can represent the whole structure of steps of the recipe. For example, if you trace arcs of a food , you can trace the cooking process of the food.

Notice: Arc label names [subj, d-obj, i-obj] in past papers are changed to [Agent, Targ, Dest] respectively.

Download

r-FG corpus Ver.2.0 (under construction)

This is a toolset for r-FG corpus. Note that this toolset does not include original recipe sentences but only annotation information. You can get original recipe sentences from "Cookpad Data Set". You can make r-FG corpus and r-NE corpus by using Both of them. Please see 00readme.txt in the toolset for details.

If this toolset does'nt work, please contact the author.

r-FG corpus

Steps of recipe are represented by a flow graph (r-FG/r-NE corpus does not include title and ingredients). Real files are shown in the right figure (csv format). One recipe corresponds to one file, one line corresponds to one vertex.

The followings are elements of a recipe flow graph file.

r-NE corpus

We annotated following information to the preparation of recipe text. Real files are shown in the right figure (tree format, available for PNAT). One recipe corresponds to one file, one line corresponds to one word. The followings are elements of a sentence in r-NE corpus file.

Corpus Specification

Corpus specification. (Ver 2.0)
Source#Recipe #Sentences #r-NEs #Words #Characters
r-NE corpusRandom4003,05817,63555,85284,536
Nikujaga362591,6044,6907,024
Chikuzenni11176240341
r-FG corpusRandom2088678,31623,357140,473
Latest version (Ver 2.0) is shown in the right table. r-FG corpus is a subset of r-NE corpus. We are enlarging the size. We aim to construct r-NE corpus consists of 1000 recipes and r-FG corpus consists of 500 recipes.

KUSK Dataset 2014RC

Kyoto University Smart Kitchen dataset (KUSK dataset) is a dataset of activity observation in a kitchen. KUSK dataset have 20 real-cooking videos from 20 recipe. 20 recipes , which are the data source of KUSK dataset 2014RC, are included in our corpus.


Links

Members

Past members

Reference

A Framework for Procedural Text Understanding,
Hirokuni Maeta, Tetsuro Sasada, Shinsuke Mori
IWPT, 2015.
KUSK Dataset: Toward a Direct Understanding of Recipe Text and Human Cooking Activity Information
Atsushi Hashimoto, Tetsuro Sasada, Yoko Yamakata, Shinsuke Mori, Michihiko Minoh,
CEA, 2014.
Flow Graph Corpus from Recipe Texts
Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, Tetsuro Sasada
to appear in LREC, 2014.
Structural Analysis of Cooking Preparation Steps in Japanese.
Reiko HAMADA, Ichiro IDE, Shuichi SAKAI, Hidehiko TANAKA.
The fifth International Workshop on Information Retrieval with Asian Languages, pp. 157–164. 2000.
Control Structures for Actions in Procedural Texts and PT-Chart
Yoshio Momouchi.
The 8th International Conference on Computational Linguistics, Vol.1, pp. 108–114. 1980.

Acknowledgements

In this research, we used recipe data provided by Cookpad and the National Institute of Informatics.

Change log

2015/07/01: Arc labels [subj, d-obj, i-obj] are changed to [Agent、Targ、Dest].
Last Change: 05/12/2016 by Tetsuro Sasada
Kyoto University ACCMS NLP Group