CLARIN-NL - text processing

OpenConvert

Jan Odijk — Tue, 21 Apr 2015 12:29:10 +0000

Title

Summary

The OpenConvert tools convert to TEI or FOLiA from a number of input formats (alto, text, word, HTML, ePub). The tools are available as a Java command line tool, a web service and a web application.

Background

The OpenConvert Tools were created by INL in the OpenConvert project. The OpenConvert tools convert to TEI or FOLiA from a number of input formats (alto, text, word, HTML, ePub). The tools are available as a Java command line tool, a web service and a web application. Furthermore, as a proof of concept, the website currently provides two annotation tools: a simple Tokenizer for TEI files and a modern Dutch part of speech tagger.

Contacts

Project leader: Jan Theo Bakker 
CLARIN center:
Help contact : helpdesk@clarin.nl

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://openconvert.clarin.inl.nl/openconvert/web/help.html
Tool/Service link:
- web application: http://openconvert.clarin.inl.nl/
- web services: see http://openconvert.clarin.inl.nl/openconvert/web/help.html
- command line tools: https://github.com/INL/OpenConvert
Publications: n.a.

Research domain

Religion Studies

Communication & Media Studies

Computational Linguistics

Cultural Sciences

History

Literary Studies

Philosophy

Political Studies

Linguistics

Language

Dutch

Language independent

Country

Netherlands

PaQu

Erica Renckens — Thu, 15 Jan 2015 13:00:04 +0000

PaQu - Parse and Query

Summary

The PaQu web service makes it possible to search in syntactically annotated corpora in Dutch. You can parse your own Dutch text corpus or use one of two corpora provided by the developers.

Background

Please note: This is a preliminary version of this application. Use Lassy (http://dev.clarin.nl/node/1966) for searching the Lassy Treebanks.

PaQu uses the Alpino parser to make treebanks of your own text corpus, and to search in these treebanks using an interface based on the LASSY Word Relations Search interface (http://dev.clarin.nl/node/1966). Two treebanks are already available in the application: Lassy Klein (1M words, manually checked syntactic analysis) and Lassy Groot (700M words, syntactic analysis automatically assigned by Alpino).
PaQu offers two ways to search through the syntactically annotated texts. The first option is to use the search bar to look for word pairs, optionally complemented by their syntactic relationship. The second search option is to use the query language XPath.

Contacts

Project leader: prof.dr. Gertjan van Noord  
Developer: Drs. P. Kleiweg (Groningen University)
CLARIN center: none
Help contact : p.c.j.kleiweg@rug.nl

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://www.let.rug.nl/alfa/paqu/info.html (http://zardoz.service.rug.nl:8067/info.html )
Tool/Service link: http://www.let.rug.nl/alfa/paqu (http://zardoz.service.rug.nl:8067/)
Publications: n.a.

Research domain

Computational Linguistics

Linguistics

Syntax

Language

Dutch

Resource tags

data

corpus

text corpus

service

text processing

mono-lingual

Tool task

annotation

corpus exploration

browse

text processing

grammatical relation assignment

parsing

Country

Netherlands

LASSY Word Relations Search

Jan Odijk — Tue, 17 Jun 2014 09:41:35 +0000

LASSY Word Relations Search Web Application

Summary

The LASSY word relations web application makes it possible to search for sentences that contain pairs of words between which there is a grammatical relation. One can search in the Dutch LASSY-SMALL Treebank (1 million tokens), in which the syntactic parse of each sentence has been manually verified, and in (a part of) the LASSY-LARGE Treebank (700 million tokens ),in which the syntactic parse of each sentence has been added by the automatic parser Alpino.

Background

One can restrict the query to search for words of a particular Part-of-Speech, which is very useful in the case of syntactic ambiguities. One can also leave out the string of the word, so that one can obtain e.g. a list of sentences in which any adverb modifies a given verb, or even any word modifies a given verb. On the page that lists the found sentences one can view the exact syntactic structure of each sentence by a simple click. The application also provides detailed frequency information of all found sentences and word pairs.
The Lassy treebanks have been made by the KU Leuven and the Rijksuniversiteit Groningen through financing of the Dutch Language Union. One can obtain these treebanks through the HLT Agency (TST-Centrale).
Use PaQu (http://dev.clarin.nl/node/4182) if you want to search for word pairs in your own text corpus.

Contacts

Project leader:  Gertjan van Noord
CLARIN center: none
Help contact : helpdesk@clarin.nl

Links

Web-sites: http://www.let.rug.nl/~alfa/lassy/bin/lassy
User scenario's (screencasts, screenshots): http://www.let.rug.nl/~alfa/lassy/
Manual: http://www.let.rug.nl/~alfa/lassy/
Tool/Service link: http://www.let.rug.nl/~alfa/lassy/bin/lassy
Publications: see http://www.let.rug.nl/vannoord/Lassy/

Research domain

Linguistics

Syntax

Language

Dutch

Resource tags

service

text processing

web-application

mono-lingual

Tool task

analysis

corpus exploration

Country

Netherlands

TTNWW

root — Sun, 15 Jun 2014 13:01:54 +0000

TTNWW - TST Tools voor het Nederlands als Webservices in een Workflow

Summary

TTNWW integrates and makes available existing Language Technology (LT) software components for the Dutch language that have been developed in the STEVIN and CGN projects. The LT components (for text and speech) are made available as web-services in a simplified workflow system that enables researchers without much technical background to use standard LT workflow recipes.

Background

The web services are available in two separate domains: "Text" and "Speech" processing. For "Text", workflows for the following functionality is offered by TTNWW:

Orthographic Normalisation using TICCLops (version CLARIN-NL 1.0)
Part of Speech Tagging, Lemmatisation, Chunking, limited Multiword Unit Recognition, and Grammatical Relation Assignment by Frog (Version 012.012)
Syntactic Parsing (including grammatical relation assignment, limited named entity recognition, and limited multiword unit recognition) by the Alpino Parser (version 1.3)
Semantic Annotation
Named Entity Recognition
Co-reference Assignment

For "Speech", the following workflows are offered:

Automatic Transcription of speech files using a Netherlands Dutch acoustic model
Automatic Transcription of speech files using a Flemish Dutch acoustic model
Conversion of the input speech file to the required sampling rate, followed by automatic transcription

The TTNWW services have been created in a Dutch and Flemish collaboration project building on the results of past Dutch and Flemish projects. The web services are partly deployed in the SURF-SARA BiG-Grid cloud or at CLARIN centres in the Netherlands and at CLARIN VL University partners.

The architecture of the TTNWW portal consists out of several components and follows the principles of Service Oriented Architecture (SOA). The TTNWW GUI front-end is a Flex module that communicates with the TTNWW web-application which keeps track of the different sessions and knows which LT recipes are available. TTNWW communicates assigments (workflow specifications) to the WorkflowService that evaluates the requested workflow and requests the DeploymentSevice to start the required LT web-services. After initialization of the LT web-services, the workflow specification is sent to the Taverna Server, that takes further care of the workflow.

To facilitate the process of wrapping applications that were originally designed as standalone applications into web services, the CLAM (Computational Linguistics Application Mediator) wrapper software allows for easy and transparent transformation of applications into RESTful web services. The CLAM software has extensively been used in the TTNWW project for both text and speech processing tools. With the exception of Alpino and MBSRL all web services work operate on CLAM wrappers.

Given the number of web services involved in the TTNWW project and possibilities offered by the cloud environment the preferred method of delivering the web service installations was delivery of complete virtual machine images by the LT providers. These could be directly uploaded into the cloud environment and thus relieving the CLARIN centres nd LT providers from the original foreseen task of running the webservices themselves. A potential advantage of this method, that has not been exploited in the project yet, is that these images may be also be delivered directly to the end user so these can be run in a local configuration using virtualization software such as VMWare of VirtualBox.

The workflow engine used in the project was Taverna. But build on top of this was a a number of selectable task recipes, following a task oriented approach in line with the premises that users with no or little technical expertise should be able to use the system. In this context, tasks are understood in terms of end results of processes such as semantic role labelling, pos tagging or syntactic analysis and ready-made workflows are constructed that can be readily used by the end user. Contacts

Project leader: Marc Kemps-Snijders (NL), Ineke Schuurman (VL)  
CLARIN center: Meertens Institute (portal) and others (web-services).
Help contact : n.a.

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://yago.meertens.knaw.nl/apache/TTNWW/assets/TTNWW.pdf
Tool/Service link: http://yago.meertens.knaw.nl/apache/TTNWW/
Publications: n.a.

Research domain

Communication & Media Studies

History

Oral History

Linguistics

Morpho-syntax

Orthography

Discourse

Semantics

Syntax

Language

Dutch

Resource tags

service

speech processing

text processing

web-application

web-service

Tool task

parsing

audio-visual processing

speech recognition

speech transcription

up/down sampling

text processing

chunking

co-reference assignment

grammatical relation assignment

multiword unit identification

orthographic normalisation

POS tagging

tokenization

lemmatisation

NE recognition

CLARIN centre

Meertens/HuC

Country

Netherlands

DCS

root — Mon, 09 Jun 2014 09:40:50 +0000

DCS - Digital Curation Service

Summary

The Digital Curation Service was a CLARIN task force responsible for the curation of several language resources that were not available on-line and/or endangered without a proper documentation context (adding CMDI metadata etc.). The DCS has curated these data and made them available by depositing them at different CLARIN centra.

Background

In October 2011 the Data Curation Service (DCS) was established. The DCS was supported by CLARIN-NL until 1 January 2014. The DCS was charged with:

the curation of resources, especially those presently held by individual researchers or research groups;
assisting in the curation efforts of CLARIN centres (if and when such is desired);
advising researchers who wanted to undertake the curation of their resources themselves.

The curation of resources held by individual researchers or research groups comprised the core of the work undertaken by the DCS. and the project was run from the Centre for Language and Speech Technology (CLST) at Radboud University Nijmegen and led by Nelleke Oostdijk (head) and Henk van den Heuvel (deputy head).

The DCS was involved in the curation of the following resources:
Completed:

LESLLA (delivered in August 2013)
- CLARIN centre: MPI for Psycholinguistics
- link: http://hdl.handle.net/1839/00-37EBCC6D-04A5-4598-88E2-E0F390D5FCE1@forma...@view
IPNV Interviews with veterans (delivered in May 2013)
- CLARIN centre: DANS
- link: https://easy.dans.knaw.nl/ui/?wicket:bookmarkablePage=:nl.knaw.dans.easy...
Woordenboek Gelderse Dialecten, Rivierengebied (delivered in June 2013)
- CLARIN centre: Meertens Institute
- link: ?
Woordenboek Gelderse Dialecten, Veluwe (part) (delivered in June 2013)
- CLARIN centre: Meertens Institute
- link: ?
Curation organisation names for OpenSkos (delivered in June 2013; finalised in August 2013)
Six dialect dictionaries from Brabant (delivered in October 2013)
- CLARIN centre: Meertens Institute
- link: ?
WLD and WBD part III (delivered in December 2013)
- CLARIN centre: Meertens Institute
- link: ?
DBD/TCULT (delivered in February 2014)
- CLARIN centre: MPI for Psycholinguistics
- link: https://corpus1.mpi.nl/ds/asv/?openpath=node:84720

Unfinished (and being considered by CLARIN centres for adoption):

Roots of Etnolect (collection Linda van Meel)
Traces of Contact (collection Kofi Yakbo)

Contacts

Project leader: Nelleke Oostdijk
CLARIN center: several
Help contact : https://www.ru.nl/letteren/datacuratieservice/informatie/contact/

Links

Web-sites: http://www.ru.nl/letteren/datacuratieservice/
User scenario's (screencasts, screenshots): n.a.
Manual: n.a.
Tool/Service link: n.a.
Publications:
- Oostdijk, N. & H. van den Heuvel. 2012. Introducing the CLARIN - NL Data Curation Service. In Proceedings of the Workshop Challenges in the management of large corpora. LREC2012, Istanbul, 22 May 2012. http://www.lrec-conf.org/proceedings/lrec2012/index.html . Retrieval date: 20 March 2014.
- Heuvel, H. van den, Oostdijk, N.H.J., Sanders, E.P. & Lint, V. de (2014). Data curations by the Dutch Data Curation Service. In Proceedings CLARIN Annual Conference (pp. online). http://www.clarin.eu/sites/default/files/cac2014_submission_15_0.pdf
- Oostdijk, N., Heuvel, H. van den, Treurniet, M. (2013): The CLARIN-NL Data Curation Service. Bringing data to the foreground. International Journal of Digital Curation, Vol 8, No 2 (2013), pp. 134-145.

Resource tags

service

speech processing

text processing

Country

Netherlands

NameScape

root — Sun, 27 Apr 2014 07:52:07 +0000

NameScape: Mapping the Landscape of Names in Modern Dutch Literature

Summary

Searching and visualizing Named Entities in modern Dutch novels.

Background

The named entity (NE) tagging and resolution in NameScape enables quantitative and repeatable research where previously only guesswork and anecdotal evidence was feasible. The visualisation module enables researchers with a less technical background to draw conclusions about functions of names in literary work and help them to explore the material in search of more interesting questions (and answers). Users from other communities (sociolinguistics, sentiment analysis, …) also benefit from the NE tagged data, especially since the NE recognizer is available as a web service, enabling researchers to annotate their own research data. Datasets in NameScape (total of 1.129 books):

Corpus Sanders: A corpus of 582 Dutch novels written and published between 1970 and 2009 will.
Corpus Huygens: Consists of 22 novels manually tagged with detailed named entity information. IPR for this corpus do not allow distribution.
Corpus eBooks: Consists of 7000+ Dutch eBooks tagged automatically with basic NER features and person name Part information. IPR for this corpus do not allow distribution.
Corpus SoNaR Books: 105 Dutch books; NE tagged.
Corpus Gutenberg Dutch: Consists of 530 NE tagged TEI files converted from the Epub versions of the corresponding Gutenberg documents.

Contacts

Project leader:  dr. Karina van Dalen-Oskam (Huygens ING)
CLARIN center: Huygens ING
Help contact : karina.van.dalen@huygens.knaw.nl

Links

Web-sites: http://www.namescape.nl/
User scenario's (demo): http://dev.clarin.nl/sites/default/files/Namescape_demonstration_scenari...
Manual: http://ner.namescape.nl/namescape/web/help.html
Metadata (VLO): http://catalog.clarin.eu/vlo/search?fq=collection:Namescape:+mapping+the...
Tool/Service link:
- Named entity tagger: http://ner.namescape.nl/namescape/tagger
- Search: http://search.namescape.nl/
- Barcode browser: http://barcode-browser.namescape.nl/index.xql
- Visualiser: http://visualizer.namescape.nl/
Publications: Karina van Dalen-Oskam (2013), Nordic Noir: a background check on Inspector Van Veeteren, 31 May 2012, http://blog.namescape.nl/?p=47

Research domain

Computational Linguistics

Literary Studies

Language

Dutch

Resource tags

corpus

text corpus

service

text processing

web-application

web-service

mono-lingual

Tool task

text processing

NE recognition

CLARIN centre

Huygens ING

Country

Netherlands

INPOLDER

root — Mon, 21 Apr 2014 14:39:57 +0000

INPOLDER: Integrated Parser and Lemmatizer Dutch in Retrospect

Summary

INPOLDER (Integrated Parser and Lemmatizer of Dutch in Retrospect) provides a tool that assigns morphological tagging, lemmatization, and syntactic parsing for historical Dutch texts. It is built on the Adelheid tool (tagging and lemmatization) and Collins-Bikel statistical Parser.

Background

As an essential part of the Dutch cultural heritage, it is of vital importance that the Dutch historical record be made accessible for research into a wide range of historical and linguistic research questions. In the transition from the Middle Ages to the modern era, the Netherlands developed from speaking a diverse group of dialects (Hollandic, Brabantic, Flemish, North-eastern, Limburgian) to a country with a standard language, and there is good reason to believe that this process was an extremely dynamic one. Systematic research into these processes affecting syntax, phonology, morphology and spelling cannot be done without access to lemmatized, tagged and parsed corpora of historical Dutch. In recent years, a tagger-lemmatizer has been developed by Hans van Halteren (Adelheid, also available in the CLARIN infrastructure). INPOLDER complements these enrichment tool with a parser for historical Dutch.

The INPOLDER parser is trained using a subset of the corpus of fourteenth-century texts (Corpus van Reenen/Mulder CRM, van Reenen and Mulder, 1993; Rem, 2003) and a subset of the Drenthe corpus (DC). CRM consists of 2700 charters from 345 places of origin. The corpus was designed as representative for the local language use of Middle Dutch and to be suitable for all types of linguistic research.

Contacts

Project leader:  Prof. Dr. Ans van Kemenade (Radboud University)
CLARIN center: Meertens Institute
Help contact : gertjan.postmaATmeertens.knaw.nl (linguistic issues), marc.kemps.snijdersATmeertens.knaw.nl (tech issues)

Links

Web-sites: http://194.171.119.69/InPolderClient/inpolder.html
User scenario's (screencasts, screenshots): http://dev.clarin.nl/sites/default/files/User%20manual-demonstatiescenar...
Manual: http://dev.clarin.nl/sites/default/files/User%20manual-demonstatiescenar...
Tool/Service link: http://194.171.119.69/InPolderClient/inpolder.html
Publications: n.a.

Research domain

Linguistics

Syntax

historical linguistics

Language

Dutch

Resource tags

service

text processing

web-application

mono-lingual

Tool task

parsing

CLARIN centre

Meertens/HuC

Country

Netherlands

TICClops

root — Sun, 20 Apr 2014 14:24:40 +0000

TiCCLops: Text-Induced Corpus Clean-up online processing system

Summary

TICCL (Text Induced Corpus Clean-up) is a system that is designed to search a corpus for all existing variants of (potentially) all words occurring in the corpus. This corpus can be one text, or several, in one or more directories, located on one or more machines. TICCL creates word frequency lists, listing for each word type how often the word occurs in the corpus. These frequencies of the normalized word forms are the sum of the frequencies of the actual word forms found in the corpus. TICCL is a system that is intended to detect and correct typographical errors (misprints) and OCR errors (optical character recognition) in texts. When books or other texts are scanned from paper by a machine, that then turns these scans, i.e. images, into digital text files, errors occur. For instance, the letter combination `in' can be read as `m', and so the word `regeering' is incorrectly reproduced as `regeermg'. TICCL can be used to detect these errors and to suggest a correct form.

Background

Text-Induced Corpus Clean-up (TICCL) was developed first as a prototype at the request of the Koninklijke Bibliotheek - The Hague (KB) and reworked into a production tool according to KB specifications (currently at production version 2.0) mainly during the second half of 2008. It is a fully functional environment for processing possibly very large corpora in order to largely remove the undesirable lexical variation in them. It has provisions for various input and output formats, is flexible and robust and has very high recall and acceptable precision. As a spelling variation detection system it is to the developer’s knowledge unique in making principled use of the input text as possible source for target output canonical forms. As such it is far less domain-sensitive than other approaches: the domain is largely covered by the input text collection.

Contacts

Project leader: dr. Martin Reynaert (Tilburg University)  
CLARIN center: The Institute for Dutch Lexicology (INL)
Help contact: n.a.

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://ticclops.uvt.nl/ticclops_manual.v101.pdf
Tool/Service link: (classic CLAM interface) http://ticclops.clarin.inl.nl/ticclops/
Tool/Service link: (@PHILOSTEI interface) http://ticclops.clarin.inl.nl/philostei/
Publications:
- Reynaert, M. (2008). All, and only, the errors: More complete and consistent spelling and OCR-error correction evaluation. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco.
- Reynaert, M. (2010). Character confusion versus focus word-based correction of spelling and ocr variants in corpora. International Journal on Document Analysis and Recognition, pp 1-15, URL http://dx.doi.org/10.1007/s10032-010-0133-5, 10.1007/s10032-010-0133-5