CLARIN-NL - orthographic normalisation

CAB orthographic canonicalizer

root — Mon, 08 Apr 2019 08:39:13 +0000

Description

orthographic normalization for historical German

CLARIN National Project

not specified

CLARIN centre

BBAW: Berlin-Brandenburg Academy of Sciences and Humanities

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

German

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

Metadata

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

CAB historical text analysis

root — Mon, 08 Apr 2019 08:39:13 +0000

Description

orthographic normalization, tagging, and lemmatization for historical German

CLARIN National Project

not specified

CLARIN centre

BBAW: Berlin-Brandenburg Academy of Sciences and Humanities

Research domain

Linguistics

Linguistic Subject

Orthography

Morpho-syntax

Syntax

Tool task

orthographic normalisation

part of speech tagging

lemmatisation

Country

Germany

Tool Type

web service

Research Phase

Enriching Data

Tool status

production

Input Language

German

Access Contact

/mailto:/

Project Contact

/mailto:/

Creator Contact

/mailto:/

Source code

http://portal.clarin.nl/not%20specified

Resource

Metadata

CMDI File Link

License

not specified

Inventory Scope

global

Distribution Type

not specified

@Philostei

root — Mon, 08 Apr 2019 08:38:55 +0000

Description

The system here allows you to convert your book pages' images into editable text, presented in a particular text format called XML (eXtended Markup Language) of a particular type called Text-Encoding Initiative or TEI XML. This particular format was developed specifically for being able to mark-up or annotate the text you want to work on, i.e. to add all manner of further information to the actual text, e.g. to build a critical edition of it, which is most likely exactly what you want to do with your author's work.

CLARIN National Project

CLARIAH-CORE

CLARIN-NL

CLARIN centre

Dutch Language Institute

Research domain

General

History

Philosophy

Linguistic Subject

general linguistics

Orthography

Tool task

corpus processing

orthographic normalisation

Country

Netherlands

Tool Type

web application

Research Phase

Enriching Data

Tool status

published

Input format

image/tiff

image/vnd.djvu

application/pdf

text/plain

text/xml

text/csv

Output format

text/xml

Input Language

Dutch

English

Finnish

French

German

German (Fraktur)

Classical Greek

Modern Greek

Icelandic

Italian

Latin

Polish

Portuguese

Russian

Spanish

Swedish

Access Contact

/mailto:servicedesk@ivdnt.org/

Institute for the Dutch Language

Project Contact

dr. Martin Reynaert

Creator Contact

Dr. Arianna Betti

Documentation

Usage

Source code

not specified

Original source

http://portal.clarin.nl/node/4190

Publications

Betti, A, Reynaert, M and van den Berg, H. 2017. @PhilosTEI: Building Corpora for Philosophers. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 379–392. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.32. License: CC-BY 4.0

Resource

CMDI File Link

License

unknown

Inventory Scope

local

PICCL

root — Mon, 08 Apr 2019 08:38:55 +0000

Title

PICCL: Philosophical Integrator of Computational and Corpus Libraries

Description

PICCL is a set of workflows for corpus building through OCR, post-correction, modernization of historic language and Natural Language Processing. It combines Tesseract Optical Character Recognition, TICCL functionality and Frog functionality in a single pipeline. Tesseract offers Open Source software for optical character recognition. TICCL (Text Induced Corpus Clean-up) is a system that is designed to search a corpus for all existing variants of (potentially) all words occurring in the corpus. This corpus can be one text, or several, in one or more directories, located on one or more machines. TICCL creates word frequency lists, listing for each word type how often the word occurs in the corpus. These frequencies of the normalized word forms are the sum of the frequencies of the actual word forms found in the corpus. TICCL is a system that is intended to detect and correct typographical errors (misprints) and OCR errors (optical character recognition) in texts. When books or other texts are scanned from paper by a machine, that then turns these scans, i.e. images, into digital text files, errors occur. For instance, the letter combination `in' can be read as `m', and so the word `regeering' is incorrectly reproduced as `regeermg'. TICCL can be used to detect these errors and to suggest a correct form. Frog enriches textual documents with various linguistic annotations.

Project

CLARIN-NL

CLARIAH-CORE

Nederlab

CLARIN National Project

CLARIN-NL

CLARIAH-CORE

CLARIN centre

none yet

Research domain

Linguistics

Philosophy

Literary Studies

Religion Studies

History

Linguistic Subject

general linguistics

Orthography

Morphology

Syntax

Tool task

optical character recognition

orthographic normalisation

sentence splitting

tokenisation

dependency parsing

shallow parsing

lemmatisation

morphological analysis

named entity recognition

part of speech tagging

Country

Netherlands

Tool Type

local desktop

Research Phase

Enriching Data

Tool status

published

Input format

application/pdf

image/tiff

text/plain

text/folia+xml

image/vnd.djvu

Output format

text/folia+xml

Input Language

Dutch

Swedish

Russian

Spanish

Portuguese

English

German

French

Italian

Finnish

Modern Greek

Classical Greek

Icelandic

German (Fraktur)

Latin

Romanian

Version

v0.6.4

Access Contact

Martin Reynaert

Project Contact

/mailto:/

Creator Contact

Martin Reynaert

Maarten van Gompel

Ko van der Sloot

Documentation

Information Page

readme

releaseNotes

issueTracker

contIntegration

Source code

https://github.com/LanguageMachines/piccl

Publications

Martin Reynaert, Maarten van Gompel, Ko van der Sloot and Antal van den Bosch. 2015. PICCL: Philosophical Integrator of Computational and Corpus Libraries. Proceedings of CLARIN Annual Conference 2015, pp. 75-79. Wrocław, Poland. http://www.nederlab.nl/cms/wp-content/uploads/2015/10/Reynaert_PICCL-Philosophical-Integrator-of-Computational-and-Corpus-Libraries.pdf

Resource

LandingPage

CMDI File Link

License

GNU GPL

Inventory Scope

local

TTNWW

root — Mon, 08 Apr 2019 08:38:55 +0000

Title

TTNWW - TST Tools for the Dutch Language as Web services in a Workflow

Description

TTNWW integrates and makes available existing Language Technology (LT) software components for the Dutch language that have been developed in the STEVIN and CGN projects. The LT components (for text and speech) are made available as web-services in a simplified workflow system that enables researchers without much technical background to use standard LT workflow recipes. The web services are available in two separate domains: "Text" and "Speech" processing. For "Text", workflows for the following functionality is offered by TTNWW: - Orthographic Normalisation using TICCLops (version CLARIN-NL 1.0); - Part of Speech Tagging, Lemmatisation, Chunking, limited Multiword Unit Recognition, and Grammatical Relation Assignment by Frog (Version 012.012); - Syntactic Parsing (including grammatical relation assignment, limited named entity recognition, and limited multiword unit recognition) by the Alpino Parser (version 1.3); - Semantic Annotation; - Named Entity Recognition; - Co-reference Assignment. For "Speech", the following workflows are offered: - Automatic Transcription of speech files using a Netherlands Dutch acoustic model; - Automatic Transcription of speech files using a Flemish Dutch acoustic model; - Conversion of the input speech file to the required sampling rate, followed by automatic transcription. The TTNWW services have been created in a Dutch and Flemish collaboration project building on the results of past Dutch and Flemish projects. The web services are partly deployed in the SURF-SARA BiG-Grid cloud or at CLARIN centres in the Netherlands and at CLARIN VL University partners. The architecture of the TTNWW portal consists out of several components and follows the principles of Service Oriented Architecture (SOA). The TTNWW GUI front-end is a Flex module that communicates with the TTNWW web-application which keeps track of the different sessions and knows which LT recipes are available. TTNWW communicates assigments (workflow specifications) to the WorkflowService that evaluates the requested workflow and requests the DeploymentSevice to start the required LT web-services. After initialization of the LT web-services, the workflow specification is sent to the Taverna Server, that takes further care of the workflow. To facilitate the process of wrapping applications that were originally designed as standalone applications into web services, the CLAM (Computational Linguistics Application Mediator) wrapper software allows for easy and transparent transformation of applications into RESTful web services. The CLAM software has extensively been used in the TTNWW project for both text and speech processing tools. With the exception of Alpino and MBSRL all web services work operate on CLAM wrappers. Given the number of web services involved in the TTNWW project and possibilities offered by the cloud environment the preferred method of delivering the web service installations was delivery of complete virtual machine images by the LT providers. These could be directly uploaded into the cloud environment and thus relieving the CLARIN centres nd LT providers from the original foreseen task of running the webservices themselves. A potential advantage of this method, that has not been exploited in the project yet, is that these images may be also be delivered directly to the end user so these can be run in a local configuration using virtualization software such as VMWare of VirtualBox. The workflow engine used in the project was Taverna. But build on top of this was a a number of selectable task recipes, following a task oriented approach in line with the premises that users with no or little technical expertise should be able to use the system. In this context, tasks are understood in terms of end results of processes such as semantic role labelling, pos tagging or syntactic analysis and ready-made workflows are constructed that can be readily used by the end user.

Project

TTNWW

CLARIN National Project

CLARIN-NL

CLARIN centre

Meertens/HuC

Research domain

Linguistics

Communication and Media Studies

History

Oral History

Linguistic Subject

discourse analysis

Orthography

Semantics

Syntax

Tool task

grammatical relation assignment

coreference resolution

corpus processing

dependency parsing

lemmatisation

multiword unit identification

named entity recognition

orthographic normalisation

part of speech tagging

semantic role labeling

chunking

parsing

speech recognition

speech transcription

tokenisation

up/down sampling

Country

Netherlands

Tool Type

web application

Research Phase

Enriching Data

Tool status

withdrawn

Input format

text/plain

audio/wav

Output format

text/xml

Input Language

Dutch

Access Contact

Daan Broeder

Project Contact

/mailto:Daan.Broeder@meertens.knaw.nl/

Creator Contact

/mailto:Daan.Broeder@meertens.knaw.nl/

Documentation

TTNWW handleiding

Source code

not specified

Original source

http://portal.clarin.nl/node/1964

Publications

Kemps-Snijders, M, Schuurman, I, Daelemans, W, Demuynck, K, Desplanques, B, Hoste, V, Huijbregts, M, Martens, J-P, Paulussen, H, Pelemans, J, Reynaert, M, Vandeghinste, V, van den Bosch, A, van denHeuvel, H, van Gompel, M, van Noord, G and Wambacq, P. 2017. TTNWW to the Rescue: No Need to Know How to Handle Tools and Resources. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 83–93. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.7. License: CC-BY 4.0

Resource

CMDI File Link

License

unknown

Inventory Scope

local

TiCClops

root — Mon, 08 Apr 2019 08:38:55 +0000

Title

TiCClops: Text-Induced Corpus Clean-up online processing system

Description

TICCL (Text Induced Corpus Clean-up) is a system that is designed to search a corpus for all existing variants of (potentially) all words occurring in the corpus. This corpus can be one text, or several, in one or more directories, located on one or more machines. TICCL creates word frequency lists, listing for each word type how often the word occurs in the corpus. These frequencies of the normalized word forms are the sum of the frequencies of the actual word forms found in the corpus. TICCL is a system that is intended to detect and correct typographical errors (misprints) and OCR errors (optical character recognition) in texts. When books or other texts are scanned from paper by a machine, that then turns these scans, i.e. images, into digital text files, errors occur. For instance, the letter combination `in' can be read as `m', and so the word `regeering' is incorrectly reproduced as `regeermg'. TICCL can be used to detect these errors and to suggest a correct form. Text-Induced Corpus Clean-up (TICCL) was developed first as a prototype at the request of the Koninklijke Bibliotheek - The Hague (KB) and reworked into a production tool according to KB specifications (currently at production version 2.0) mainly during the second half of 2008. It is a fully functional environment for processing possibly very large corpora in order to largely remove the undesirable lexical variation in them. It has provisions for various input and output formats, is flexible and robust and has very high recall and acceptable precision. As a spelling variation detection system it is to the developer’s knowledge unique in making principled use of the input text as possible source for target output canonical forms. As such it is far less domain-sensitive than other approaches: the domain is largely covered by the input text collection. TICCL comes in two variants: one with a classic CLAM web application interface, and one with the PhilosTEI interface.

Project

TiCClops: Text-Induced Corpus Clean-up online processing system

CLARIN National Project

CLARIN-NL

CLARIN centre

Dutch Language Institute

Research domain

Linguistics

Linguistic Subject

general linguistics

Orthography

Tool task

corpus processing

orthographic normalisation

Country

Netherlands

Tool Type

web application

Research Phase

Enriching Data

Tool status

published

Input format

image/tiff

image/vnd.djvu

application/pdf

text/plain

text/xml

text/csv

Output format

text/xml

Input Language

Dutch

English

Finnish

French

German

German (Fraktur)

Classical Greek

Modern Greek

Icelandic

Italian

Latin

Polish

Portuguese

Russian

Spanish

Swedish

Access Contact

/mailto:servicedesk@ivdnt.org/

Institute for the Dutch Language

Project Contact

dr. Martin Reynaert

Creator Contact

dr. Martin Reynaert

Documentation

TICCLops User and Demonstrator Documentation

Source code

not specified

Original source

http://portal.clarin.nl/node/1914

Publications

Reynaert, M. (2008). All, and only, the errors: More complete and consistent spelling and OCR-error correction evaluation. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco.

Reynaert, M. (2010). Character confusion versus focus word-based correction of spelling and ocr variants in corpora. International Journal on Document Analysis and Recognition, pp 1-15, URL http://dx.doi.org/10.1007/s10032-010-0133-5

Resource

CMDI File Link

License

unknown

Inventory Scope

local

@PhilosTEI

Erica Renckens — Wed, 21 Jan 2015 15:05:45 +0000

@philosTEI - TICCLing Philosophy

Summary

An open source, web-based, user-friendly workflow from digital images of text to TEI, which enables building digital philosophy corpora. This workflow uses a combination of an OCRopus / Tesseract webservice for text layout analysis and Optical Character Recognition (OCR) and a multilingual version of TICCL available as webservice TICCLops.

Background

Philosophy and philosophy-informed intellectual history can profit immensely from teaming up with computer scientists and applying computational methods. A new empirical, datadriven and collaborative computational methodology can scale up our research and help making it more efficient and objective. The tool is able to build appropriate high quality, easily accessible, large-scale corpora of multi-script texts in a sustainable format (TEI) which are multi-language and from different historical periods in a reliable, efficient, user-friendly, sustainable and cost-cutting way.

This is demonstrated on a multi-lingual, multi-script corpus of 18th-20th century philosophical texts. This test case corpus contains at least the following books and articles (maybe more will follow):

Wolff (1740) (Latin)
Bolzano (1837) (German)
Frege (1879) (German)
Tarski (1936) (Polish)

Contacts

Project leader:  Dr. Arianna Betti (VU University Amsterdam)
CLARIN center: Huygens ING
Help contact : t.b.a.
Twitter: https://twitter.com/philostei

Links

Web-sites: http://www.axiom.humanities.uva.nl/PhilosTEI.html
User scenario's (screencasts, screenshots): See ? demo menu item on http://ticclops.clarin.inl.nl/philostei/
Manual: http://ticclops.clarin.inl.nl/philostei/about.html
Tool/Service link: http://ticclops.clarin.inl.nl/philostei/
Publications: t.b.a.

Research domain

History

Philosophy

Language

German

Multiple languages

Resource tags

data

corpus

service

web-application

multi-lingual

Tool task

text processing

orthographic normalisation

CLARIN centre

Huygens ING

Country

Netherlands

TTNWW

root — Sun, 15 Jun 2014 13:01:54 +0000

TTNWW - TST Tools voor het Nederlands als Webservices in een Workflow

Summary

Background

The web services are available in two separate domains: "Text" and "Speech" processing. For "Text", workflows for the following functionality is offered by TTNWW:

Orthographic Normalisation using TICCLops (version CLARIN-NL 1.0)
Part of Speech Tagging, Lemmatisation, Chunking, limited Multiword Unit Recognition, and Grammatical Relation Assignment by Frog (Version 012.012)
Syntactic Parsing (including grammatical relation assignment, limited named entity recognition, and limited multiword unit recognition) by the Alpino Parser (version 1.3)
Semantic Annotation
Named Entity Recognition
Co-reference Assignment

For "Speech", the following workflows are offered:

Automatic Transcription of speech files using a Netherlands Dutch acoustic model
Automatic Transcription of speech files using a Flemish Dutch acoustic model
Conversion of the input speech file to the required sampling rate, followed by automatic transcription

The TTNWW services have been created in a Dutch and Flemish collaboration project building on the results of past Dutch and Flemish projects. The web services are partly deployed in the SURF-SARA BiG-Grid cloud or at CLARIN centres in the Netherlands and at CLARIN VL University partners.

The architecture of the TTNWW portal consists out of several components and follows the principles of Service Oriented Architecture (SOA). The TTNWW GUI front-end is a Flex module that communicates with the TTNWW web-application which keeps track of the different sessions and knows which LT recipes are available. TTNWW communicates assigments (workflow specifications) to the WorkflowService that evaluates the requested workflow and requests the DeploymentSevice to start the required LT web-services. After initialization of the LT web-services, the workflow specification is sent to the Taverna Server, that takes further care of the workflow.

To facilitate the process of wrapping applications that were originally designed as standalone applications into web services, the CLAM (Computational Linguistics Application Mediator) wrapper software allows for easy and transparent transformation of applications into RESTful web services. The CLAM software has extensively been used in the TTNWW project for both text and speech processing tools. With the exception of Alpino and MBSRL all web services work operate on CLAM wrappers.

Given the number of web services involved in the TTNWW project and possibilities offered by the cloud environment the preferred method of delivering the web service installations was delivery of complete virtual machine images by the LT providers. These could be directly uploaded into the cloud environment and thus relieving the CLARIN centres nd LT providers from the original foreseen task of running the webservices themselves. A potential advantage of this method, that has not been exploited in the project yet, is that these images may be also be delivered directly to the end user so these can be run in a local configuration using virtualization software such as VMWare of VirtualBox.

The workflow engine used in the project was Taverna. But build on top of this was a a number of selectable task recipes, following a task oriented approach in line with the premises that users with no or little technical expertise should be able to use the system. In this context, tasks are understood in terms of end results of processes such as semantic role labelling, pos tagging or syntactic analysis and ready-made workflows are constructed that can be readily used by the end user. Contacts

Project leader: Marc Kemps-Snijders (NL), Ineke Schuurman (VL)  
CLARIN center: Meertens Institute (portal) and others (web-services).
Help contact : n.a.

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://yago.meertens.knaw.nl/apache/TTNWW/assets/TTNWW.pdf
Tool/Service link: http://yago.meertens.knaw.nl/apache/TTNWW/
Publications: n.a.

Research domain

Communication & Media Studies

History

Oral History

Linguistics

Morpho-syntax

Orthography

Discourse

Semantics

Syntax

Language

Dutch

Resource tags

service

speech processing

text processing

web-application

web-service

Tool task

parsing

audio-visual processing

speech recognition

speech transcription

up/down sampling

text processing

chunking

co-reference assignment

grammatical relation assignment

multiword unit identification

orthographic normalisation

POS tagging

tokenization

lemmatisation

NE recognition

CLARIN centre

Meertens/HuC

Country

Netherlands

TICClops

root — Sun, 20 Apr 2014 14:24:40 +0000

TiCCLops: Text-Induced Corpus Clean-up online processing system

Summary

Background

Text-Induced Corpus Clean-up (TICCL) was developed first as a prototype at the request of the Koninklijke Bibliotheek - The Hague (KB) and reworked into a production tool according to KB specifications (currently at production version 2.0) mainly during the second half of 2008. It is a fully functional environment for processing possibly very large corpora in order to largely remove the undesirable lexical variation in them. It has provisions for various input and output formats, is flexible and robust and has very high recall and acceptable precision. As a spelling variation detection system it is to the developer’s knowledge unique in making principled use of the input text as possible source for target output canonical forms. As such it is far less domain-sensitive than other approaches: the domain is largely covered by the input text collection.

Contacts

Project leader: dr. Martin Reynaert (Tilburg University)  
CLARIN center: The Institute for Dutch Lexicology (INL)
Help contact: n.a.

Links

Web-sites: n.a.
User scenario's (screencasts, screenshots): n.a.
Manual: http://ticclops.uvt.nl/ticclops_manual.v101.pdf
Tool/Service link: (classic CLAM interface) http://ticclops.clarin.inl.nl/ticclops/
Tool/Service link: (@PHILOSTEI interface) http://ticclops.clarin.inl.nl/philostei/
Publications:
- Reynaert, M. (2008). All, and only, the errors: More complete and consistent spelling and OCR-error correction evaluation. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco.
- Reynaert, M. (2010). Character confusion versus focus word-based correction of spelling and ocr variants in corpora. International Journal on Document Analysis and Recognition, pp 1-15, URL http://dx.doi.org/10.1007/s10032-010-0133-5, 10.1007/s10032-010-0133-5

Research domain

Linguistics

Orthography

Language

Dutch

Resource tags

service

text processing

web-application

Tool task

text processing

orthographic normalisation

CLARIN centre

Dutch Language Institute

Country

Netherlands