Text data in Corpus and other packages

Text data type

The corpus package does not define a special corpus object, but it does define a new data type, corpus_text, for storing a collection of texts. You can create values of this type using the as_corpus_text() or as_corpus_frame() function.

Take, for example, the following sample text, created as an R character vector.

# raw text for the first two paragraphs of _The Tale of Peter Rabbit_,
# by Beatrix Potter
raw <- c(
    para1 =
        paste("Once upon a time there were four little Rabbits,",
              "and their names were: Flopsy, Mopsy, Cottontail, and Peter.",
              "They lived with their Mother in a sandbank,",
              "underneath the root of a very big fir tree.",
              sep = "\n"),
    para2 =
        paste("'Now, my dears,' said old Mrs. Rabbit one morning,",
              "'you may go into the fields or down the lane,",
              "but don't go into Mr. McGregor's garden --",
              "your Father had an accident there;",
              "he was put in a pie by Mrs. McGregor.'",
              sep = "\n"))

We can convert the text to a corpus_text object using the as_corpus_text() function:

text <- as_corpus_text(raw)

Alternatively, we can convert it to a data frame with a column named "text" of type corpus_text using the as_corpus_frame() function:

data <- as_corpus_frame(raw)

Both as_corpus_frame() and as_corpus_text() are generic; they work on character vectors, data frames, tm Corpus objects, and quanteda corpus objects. Whenever you call a corpus function expecting text, that function calls as_corpus_text() on its input. We will see some examples of converting tm and quanteda objects below.

The corpus_text type behaves like an R character vector in most respects, but using this type enables some new features.

Text filter

Each corpus_text object has a text_filter property that can be get or set using the text_filter() generic function. This property allows us to specify the preprocessing decisions that define the text normalization, token, and sentence boundaries.

# get the text filter
print(text_filter(text))
Text filter with the following options:

    map_case: TRUE
    map_quote: TRUE
    remove_ignorable: TRUE
    combine: NULL
    stemmer: NULL
    stem_dropped: FALSE
    stem_except: NULL
    drop_letter: FALSE
    drop_number: FALSE
    drop_punct: FALSE
    drop_symbol: FALSE
    drop: NULL
    drop_except: NULL
    connector: _
    sent_crlf: FALSE
    sent_suppress:  chr [1:155] "A." "A.D." "a.m." "A.M." "A.S." "AA." "AB." "Abs." "AD." ...
# set a new text filter
text_filter(text) <- text_filter(drop_punct = TRUE, drop = stopwords_en)

# update a text filter property
text_filter(text)$drop_punct <- TRUE

# switch to the default text filter
text_filter(text) <- NULL

Valid UTF-8

All corpus_text objects contain valid UTF-8 data. It is impossible to create a corpus_text object with invalid UTF-8:

# the input is encoded in Latin-1, but declared as UTF-8
x <- "fa\xE7ile"
Encoding(x) <- "UTF-8"
as_corpus_text(x) ## fails
Error in as_corpus_text.character(x): argument entry 1 is incorrectly marked as "UTF-8": leading byte 0xE7 followed by invalid continuation byte (0x69) at position 4
as_corpus_text(iconv(x, "Latin1", "UTF-8")) ## succeeds
[1] "façile"

Unique names

All corpus_text objects have unique names. When you coerce an object with duplicate names to corpus_text, the duplicates get renamed with a warning:

x <- as_corpus_text(c(a = "hello", b = "world", a = "repeated name")) # fails
Warning in as_corpus_text.character(c(a = "hello", b = "world", a = "repeated name")): renaming
entries with duplicate names
names(x)
[1] "a"   "b"   "a.1"

Setting repeated names fails:

# fails, duplicate names
names(x) <- c("a", "b", "a")
Error in `names<-.corpus_text`(`*tmp*`, value = c("a", "b", "a")): duplicate 'names' are not allowed

You can set the names to NULL if you don’t want them:

names(x) <- NULL
print(names(x))
NULL

Reference memory-mapped data

corpus_text objects can manage collections of texts that do not fit into memory, through the memory-map interface provided by read_ndjson(). When dealing with such objects, internally the corpus_text object just stores an offset into the file containing the text, not the text itself.

# store some sample data in a newline-delimited JSON file
tmp <- tempfile()
writeLines(c('{"text": "A sample text", "metadata": 7 }',
             '{"text": "Another text."}',
             '{"text": "A third text.", "metadata": -100}'),
           tmp)

# memory-map the "text" field
data <- read_ndjson(tmp, text = "text", mmap = TRUE)

# display the internal representation of the resulting 'corpus_text' object
unclass(data$text)
$handle
<pointer: 0x7fba65dd4ba0>

$sources
$sources[[1]]
JSON data set with 3 rows of type text


$table
  source row start stop
1      1   1     1   13
2      1   2     1   13
3      1   3     1   13

$names
NULL

$filter
NULL

Above, the handle component of the text object is a memory-mapped view of the file. The table component stores the row numbers for the data, along with the start and stop indices of the text.

Reference substrings of other objects

corpus_text objects can reference substrings of other R character objects, without copying the parent object. This allows functions like text_split() and text_locate() to quickly split a text into smaller segments without allocating new character strings.

# segment text into chunks of at most 10 tokens
(chunks <- text_split(text, "tokens", 10))
   parent index text                                           
1  para1      1 Once upon a time there were four little Rabbits
2  para1      2 ,\nand their names were: Flopsy, Mopsy         
3  para1      3 , Cottontail, and Peter.\nThey lived with      
4  para1      4 their Mother in a sandbank,\nunderneath the    
5  para1      5 root of a very big fir tree.                   
6  para2      1 'Now, my dears,' said old Mrs                  
7  para2      2 . Rabbit one morning,\n'you may go into        
8  para2      3 the fields or down the lane,\nbut don't        
9  para2      4 go into Mr. McGregor's garden --\nyour         
10 para2      5 Father had an accident there;\nhe was put      
11 para2      6 in a pie by Mrs. McGregor.'                    
# segmenting does not allocate new objects for the segments
unclass(chunks$text) # inspect text internals
$handle
<pointer: 0x7fba65fa0fb0>

$sources
$sources[[1]]
                                                                                                                                                                                                                      para1 
                  "Once upon a time there were four little Rabbits,\nand their names were: Flopsy, Mopsy, Cottontail, and Peter.\nThey lived with their Mother in a sandbank,\nunderneath the root of a very big fir tree." 
                                                                                                                                                                                                                      para2 
"'Now, my dears,' said old Mrs. Rabbit one morning,\n'you may go into the fields or down the lane,\nbut don't go into Mr. McGregor's garden --\nyour Father had an accident there;\nhe was put in a pie by Mrs. McGregor.'" 


$table
   source row start stop
1       1   1     1   47
2       1   1    48   84
3       1   1    85  125
4       1   1   126  168
5       1   1   169  196
6       1   2     1   29
7       1   2    30   68
8       1   2    69  107
9       1   2   108  145
10      1   2   146  186
11      1   2   187  213

$names
NULL

$filter
NULL

The call to text_split() did not allocate any new character objects; the result uses the same sources, but with different start and stop indices.

Better printing

Printing corpus_text objects truncates the output at the end of the line instead of printing the entire contents.

# print a character object
print(raw)
                                                                                                                                                                                                                      para1 
                  "Once upon a time there were four little Rabbits,\nand their names were: Flopsy, Mopsy, Cottontail, and Peter.\nThey lived with their Mother in a sandbank,\nunderneath the root of a very big fir tree." 
                                                                                                                                                                                                                      para2 
"'Now, my dears,' said old Mrs. Rabbit one morning,\n'you may go into the fields or down the lane,\nbut don't go into Mr. McGregor's garden --\nyour Father had an accident there;\nhe was put in a pie by Mrs. McGregor.'" 
# print a corpus_text object
print(text)
para1                                                                                          
"Once upon a time there were four little Rabbits,\nand their names were: Flopsy, Mopsy, Cotto…"
para2                                                                                          
"'Now, my dears,' said old Mrs. Rabbit one morning,\n'you may go into the fields or down the …"

If you’d like to print the entire contents, you can convert the text to character and then print:

# print entire contents
print(as.character(text))
[1] "Once upon a time there were four little Rabbits,\nand their names were: Flopsy, Mopsy, Cottontail, and Peter.\nThey lived with their Mother in a sandbank,\nunderneath the root of a very big fir tree."                  
[2] "'Now, my dears,' said old Mrs. Rabbit one morning,\n'you may go into the fields or down the lane,\nbut don't go into Mr. McGregor's garden --\nyour Father had an accident there;\nhe was put in a pie by Mrs. McGregor.'"

Conversion to character

The one disadvantage of using a corpus_text object instead of an R character vector is that many functions from other packages that expect R character vectors will not work on corpus_text. To get around this, you can call as.character on a corpus_text object to convert it to an R character vector.

# some R methods coerce their inputs to character; 'message' is one example
message(as_corpus_text("hello world"))
hello world
# others do not;
cat(as_corpus_text("hello world"), "\n") # fails
Error in cat(as_corpus_text("hello world"), "\n"): argument 1 (type 'list') cannot be handled by 'cat'
# we must call 'as.character' on the inputs
cat(as.character(as_corpus_text("hello world")), "\n")
hello world 

Interface with quanteda

All corpus functions expecting text work on quanteda corpus objects:

uk2010immigCorpus <- 
    quanteda::corpus(quanteda::data_char_ukimmig2010,
           docvars = data.frame(party = names(quanteda::data_char_ukimmig2010)),
           metacorpus = list(notes = "Immigration-related sections of 2010 UK party manifestos"))

# search for terms in a quanteda corpus that stem to "immigr":
text_locate(uk2010immigCorpus, "immigr", stemmer = "en")
   text                before                  instance                   after                
1  BNP                                        IMMIGRATION : AN UNPARALLELED CRISIS WHICH ONLY …
2  BNP  …THE BNP CAN SOLVE. \n\n- At current  immigration  and birth rates, indigenous British…
3  BNP  … will include a halt to all further  immigration , the deportation of all illegal imm…
4  BNP  …ion, the deportation of all illegal  immigrants  , a halt to the "asylum" swindle and…
5  BNP  …mes in Britain, regardless of their  immigration  status.\n\n- The BNP will review al…
6  BNP  …mission that they orchestrated mass  immigration  to change forcibly Britain's demogr…
7  BNP  …ce is in grave peril, threatened by  immigration  and multiculturalism. In the absenc…
8  BNP  …Statistics (ONS), legal Third World  immigrants   made up 14.7 percent (7.5 million) …
9  BNP  …rths to second and third generation   immigrant   mothers. Figures released by the ON…
10 BNP  …hen these figures are added in, the   immigrant   birth rate is estimated to be aroun…
11 BNP  …ales.\n\n- The majority of the 'new  immigrants  ' are not from Eastern Europe, as is…
12 BNP  …imed. According to the ONS figures,  immigrants   from Eastern Europe had 25,000 chil…
13 BNP  …er exponentially, and given current  immigration  and birth rates, will utterly overw…
14 BNP  …s.\n\nThe Disastrous Effect of Mass  Immigration  on British Society\n\nThere is no e…
15 BNP  …here that it is 'racist' to discuss  immigration  and population density.\n\nThe word…
16 BNP  …between 300,000-500,000 Third World  immigrants   each year is an issue that all thre…
17 BNP  …d so on.\n\nA Case Study: Crime and  Immigration \n\nImmigration has had a dramatic e…
18 BNP  …ase Study: Crime and Immigration\n\n Immigration  has had a dramatic effect on Britai…
19 BNP  …onately involving ethnic groups.\n\n Immigration  Has Harmed British Jobs\n\nThe conc…
20 BNP  …s working than in 1997.\n\nOverall,  immigrants   have taken up more than 1.64 millio…
⋮  (87 rows total)

You can convert a quanteda corpus to a data frame using as_corpus_frame(), or extract its text using as_corpus_text():

# convert to corpus_text:
print(as_corpus_text(uk2010immigCorpus))
BNP                                                                                            
"IMMIGRATION: AN UNPARALLELED CRISIS WHICH ONLY THE BNP CAN SOLVE. \n\n- At current immigrati…"
Coalition                                                                                      
"IMMIGRATION. \n\nThe Government believes that immigration has enriched our culture and stren…"
Conservative                                                                                   
"Attract the brightest and best to our country.\n\nImmigration has enriched our nation over t…"
Greens                                                                                         
"Immigration.\n\nMigration is a fact of life.  People have always moved from one country to a…"
Labour                                                                                         
"Crime and immigration\n\nThe challenge for Britain\n\nWe will control immigration with our n…"
LibDem                                                                                         
"firm but fair immigration system\n\nBritain has always been an open, welcoming country, and …"
PC                                                                                             
"As a welcoming nation, Plaid Cymru recognises the invaluable contribution that migrants have…"
SNP                                                                                            
"And we will argue for Scotland to take responsibility for immigration so that we can develop…"
UKIP                                                                                           
"Immigration & Asylum.\n\nAs a member of the EU, Britain has lost control of her borders. Som…"
# convert to data frame:
print(as_corpus_frame(uk2010immigCorpus))
             party        text                                                                 
BNP          BNP          IMMIGRATION: AN UNPARALLELED CRISIS WHICH ONLY THE BNP CAN SOLVE. \n…
Coalition    Coalition    IMMIGRATION. \n\nThe Government believes that immigration has enrich…
Conservative Conservative Attract the brightest and best to our country.\n\nImmigration has en…
Greens       Greens       Immigration.\n\nMigration is a fact of life.  People have always mov…
Labour       Labour       Crime and immigration\n\nThe challenge for Britain\n\nWe will contro…
LibDem       LibDem       firm but fair immigration system\n\nBritain has always been an open,…
PC           PC           As a welcoming nation, Plaid Cymru recognises the invaluable contrib…
SNP          SNP          And we will argue for Scotland to take responsibility for immigratio…
UKIP         UKIP         Immigration & Asylum.\n\nAs a member of the EU, Britain has lost con…

Interface with tm

All corpus functions expecting text also work on tm Corpus objects:

data("crude", package = "tm")

# get the top terms in a tm corpus:
term_stats(crude, drop_punct = TRUE, drop = stopwords_en)
   term      count support
1  oil          85      20
2  said         73      20
3  reuter       20      20
4  prices       48      15
5  last         24      12
6  one          17      12
7  dlrs         23      11
8  opec         44      10
9  barrel       15      10
10 mln          31       9
11 crude        21       9
12 new          14       9
13 pct          14       9
14 price        13       9
15 also          9       9
16 petroleum     9       9
17 market       20       8
18 barrels      11       8
19 industry     10       8
20 world        10       8
⋮  (1042 rows total)

The as_corpus_frame() and as_corpus_text() functions also work on tm Corpus objects:

# convert to corpus_text
print(as_corpus_text(crude))
127                                                                                            
"Diamond Shamrock Corp said that\neffective today it had cut its contract prices for crude oi…"
144                                                                                            
"OPEC may be forced to meet before a\nscheduled June session to readdress its production cutt…"
191                                                                                            
"Texaco Canada said it lowered the\ncontract price it will pay for crude oil 64 Canadian cts …"
194                                                                                            
"Marathon Petroleum Co said it reduced\nthe contract price it will pay for all grades of crud…"
211                                                                                            
"Houston Oil Trust said that independent\npetroleum engineers completed an annual study that …"
236                                                                                            
"Kuwait\"s Oil Minister, in remarks\npublished today, said there were no plans for an emergen…"
237                                                                                            
"Indonesia appears to be nearing a\npolitical crossroads over measures to deregulate its prot…"
242                                                                                            
"Saudi riyal interbank deposits were\nsteady at yesterday's higher levels in a quiet market.…" 
246                                                                                            
"The Gulf oil state of Qatar, recovering\nslightly from last year's decline in world oil pric…"
248                                                                                            
"Saudi Arabian Oil Minister Hisham Nazer\nreiterated the kingdom's commitment to last Decembe…"
273                                                                                            
"Saudi crude oil output last month fell\nto an average of 3.5 mln barrels per day (bpd) from …"
349                                                                                            
"Deputy oil ministers from six Gulf\nArab states will meet in Bahrain today to discuss coordi…"
352                                                                                            
"Saudi Arabian Oil Minister Hisham Nazer\nreiterated the kingdom's commitment to last Decembe…"
353                                                                                            
"Kuwait's oil minister said in a newspaper\ninterview that there were no plans for an emergen…"
368                                                                                            
"The port of Philadelphia was closed\nwhen a Cypriot oil tanker, Seapride II, ran aground aft…"
489                                                                                            
"A study group said the United States\nshould increase its strategic petroleum reserve to one…"
502                                                                                            
"A study group said the United States\nshould increase its strategic petroleum reserve to one…"
543                                                                                            
"Unocal Corp's Union Oil Co said it\nlowered its posted prices for crude oil one to 1.50 dlrs…"
704                                                                                            
"The New York Mercantile Exchange set\nApril one for the debut of a new procedure in the ener…"
708                                                                                            
"Argentine crude oil production was\ndown 10.8 pct in January 1987 to 12.32 mln barrels, from…"
# convert to data frame
print(as_corpus_frame(crude))
    text                                                                                       
127 Diamond Shamrock Corp said that\neffective today it had cut its contract prices for crude …
144 OPEC may be forced to meet before a\nscheduled June session to readdress its production cu…
191 Texaco Canada said it lowered the\ncontract price it will pay for crude oil 64 Canadian ct…
194 Marathon Petroleum Co said it reduced\nthe contract price it will pay for all grades of cr…
211 Houston Oil Trust said that independent\npetroleum engineers completed an annual study tha…
236 Kuwait"s Oil Minister, in remarks\npublished today, said there were no plans for an emerge…
237 Indonesia appears to be nearing a\npolitical crossroads over measures to deregulate its pr…
242 Saudi riyal interbank deposits were\nsteady at yesterday's higher levels in a quiet market…
246 The Gulf oil state of Qatar, recovering\nslightly from last year's decline in world oil pr…
248 Saudi Arabian Oil Minister Hisham Nazer\nreiterated the kingdom's commitment to last Decem…
273 Saudi crude oil output last month fell\nto an average of 3.5 mln barrels per day (bpd) fro…
349 Deputy oil ministers from six Gulf\nArab states will meet in Bahrain today to discuss coor…
352 Saudi Arabian Oil Minister Hisham Nazer\nreiterated the kingdom's commitment to last Decem…
353 Kuwait's oil minister said in a newspaper\ninterview that there were no plans for an emerg…
368 The port of Philadelphia was closed\nwhen a Cypriot oil tanker, Seapride II, ran aground a…
489 A study group said the United States\nshould increase its strategic petroleum reserve to o…
502 A study group said the United States\nshould increase its strategic petroleum reserve to o…
543 Unocal Corp's Union Oil Co said it\nlowered its posted prices for crude oil one to 1.50 dl…
704 The New York Mercantile Exchange set\nApril one for the debut of a new procedure in the en…
708 Argentine crude oil production was\ndown 10.8 pct in January 1987 to 12.32 mln barrels, fr…